aspera-cli 4.6.0 → 4.7.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (96) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +427 -300
  3. data/bin/ascli +2 -1
  4. data/bin/asession +1 -0
  5. data/docs/test_env.conf +2 -0
  6. data/examples/aoc.rb +4 -3
  7. data/examples/faspex4.rb +21 -19
  8. data/examples/proxy.pac +1 -1
  9. data/examples/transfer.rb +15 -15
  10. data/lib/aspera/aoc.rb +135 -124
  11. data/lib/aspera/ascmd.rb +85 -75
  12. data/lib/aspera/ats_api.rb +11 -10
  13. data/lib/aspera/cli/basic_auth_plugin.rb +13 -14
  14. data/lib/aspera/cli/extended_value.rb +42 -33
  15. data/lib/aspera/cli/formater.rb +138 -111
  16. data/lib/aspera/cli/info.rb +17 -0
  17. data/lib/aspera/cli/listener/line_dump.rb +3 -2
  18. data/lib/aspera/cli/listener/logger.rb +2 -1
  19. data/lib/aspera/cli/listener/progress.rb +16 -18
  20. data/lib/aspera/cli/listener/progress_multi.rb +13 -16
  21. data/lib/aspera/cli/main.rb +122 -130
  22. data/lib/aspera/cli/manager.rb +146 -154
  23. data/lib/aspera/cli/plugin.rb +38 -34
  24. data/lib/aspera/cli/plugins/alee.rb +6 -6
  25. data/lib/aspera/cli/plugins/aoc.rb +273 -276
  26. data/lib/aspera/cli/plugins/ats.rb +82 -76
  27. data/lib/aspera/cli/plugins/bss.rb +14 -16
  28. data/lib/aspera/cli/plugins/config.rb +350 -306
  29. data/lib/aspera/cli/plugins/console.rb +23 -19
  30. data/lib/aspera/cli/plugins/cos.rb +18 -18
  31. data/lib/aspera/cli/plugins/faspex.rb +180 -159
  32. data/lib/aspera/cli/plugins/faspex5.rb +64 -54
  33. data/lib/aspera/cli/plugins/node.rb +147 -140
  34. data/lib/aspera/cli/plugins/orchestrator.rb +68 -66
  35. data/lib/aspera/cli/plugins/preview.rb +92 -96
  36. data/lib/aspera/cli/plugins/server.rb +79 -75
  37. data/lib/aspera/cli/plugins/shares.rb +23 -24
  38. data/lib/aspera/cli/plugins/sync.rb +20 -22
  39. data/lib/aspera/cli/transfer_agent.rb +40 -39
  40. data/lib/aspera/cli/version.rb +2 -1
  41. data/lib/aspera/colors.rb +35 -27
  42. data/lib/aspera/command_line_builder.rb +48 -34
  43. data/lib/aspera/cos_node.rb +29 -21
  44. data/lib/aspera/data_repository.rb +3 -2
  45. data/lib/aspera/environment.rb +50 -45
  46. data/lib/aspera/fasp/agent_base.rb +22 -20
  47. data/lib/aspera/fasp/agent_connect.rb +13 -11
  48. data/lib/aspera/fasp/agent_direct.rb +48 -59
  49. data/lib/aspera/fasp/agent_httpgw.rb +33 -39
  50. data/lib/aspera/fasp/agent_node.rb +15 -13
  51. data/lib/aspera/fasp/agent_trsdk.rb +12 -14
  52. data/lib/aspera/fasp/error.rb +2 -1
  53. data/lib/aspera/fasp/error_info.rb +68 -52
  54. data/lib/aspera/fasp/installation.rb +106 -94
  55. data/lib/aspera/fasp/listener.rb +1 -0
  56. data/lib/aspera/fasp/parameters.rb +83 -92
  57. data/lib/aspera/fasp/parameters.yaml +305 -249
  58. data/lib/aspera/fasp/resume_policy.rb +11 -14
  59. data/lib/aspera/fasp/transfer_spec.rb +26 -0
  60. data/lib/aspera/fasp/uri.rb +22 -21
  61. data/lib/aspera/faspex_gw.rb +55 -90
  62. data/lib/aspera/hash_ext.rb +4 -3
  63. data/lib/aspera/id_generator.rb +8 -7
  64. data/lib/aspera/keychain/encrypted_hash.rb +17 -16
  65. data/lib/aspera/keychain/macos_security.rb +6 -10
  66. data/lib/aspera/log.rb +25 -20
  67. data/lib/aspera/nagios.rb +13 -12
  68. data/lib/aspera/node.rb +30 -22
  69. data/lib/aspera/oauth.rb +175 -226
  70. data/lib/aspera/open_application.rb +4 -3
  71. data/lib/aspera/persistency_action_once.rb +6 -6
  72. data/lib/aspera/persistency_folder.rb +5 -9
  73. data/lib/aspera/preview/file_types.rb +6 -5
  74. data/lib/aspera/preview/generator.rb +25 -24
  75. data/lib/aspera/preview/options.rb +16 -14
  76. data/lib/aspera/preview/utils.rb +98 -98
  77. data/lib/aspera/{proxy_auto_config.erb.js → proxy_auto_config.js} +23 -31
  78. data/lib/aspera/proxy_auto_config.rb +111 -20
  79. data/lib/aspera/rest.rb +115 -113
  80. data/lib/aspera/rest_call_error.rb +2 -2
  81. data/lib/aspera/rest_error_analyzer.rb +23 -25
  82. data/lib/aspera/rest_errors_aspera.rb +15 -14
  83. data/lib/aspera/ssh.rb +12 -10
  84. data/lib/aspera/sync.rb +42 -41
  85. data/lib/aspera/temp_file_manager.rb +18 -14
  86. data/lib/aspera/timer_limiter.rb +2 -1
  87. data/lib/aspera/uri_reader.rb +7 -5
  88. data/lib/aspera/web_auth.rb +79 -76
  89. metadata +64 -21
  90. data/docs/Makefile +0 -65
  91. data/docs/README.erb.md +0 -4424
  92. data/docs/README.md +0 -13
  93. data/docs/diagrams.txt +0 -49
  94. data/docs/doc_tools.rb +0 -58
  95. data/lib/aspera/cli/plugins/shares2.rb +0 -114
  96. data/lib/aspera/fasp/default.rb +0 -17
data/docs/README.erb.md DELETED
@@ -1,4424 +0,0 @@
1
- # Command Line Interface for IBM Aspera products
2
-
3
- [comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
4
- <%load File.join(File.dirname(__FILE__),'doc_tools.rb')-%>
5
-
6
- Version : <%=gemspec.version.to_s%>
7
-
8
- Laurent/2016-<%=Time.new.year%>
9
-
10
- This gem provides the <%=tool%> Command Line Interface to IBM Aspera software.
11
-
12
- <%=tool%> is a also great tool to learn Aspera APIs.
13
-
14
- Ruby Gem: [<%=gemspec.metadata['rubygems_uri']%>](<%=gemspec.metadata['rubygems_uri']%>)
15
-
16
- Ruby Doc: [<%=gemspec.metadata['documentation_uri']%>](<%=gemspec.metadata['documentation_uri']%>)
17
-
18
- Required Ruby version: <%=gemspec.required_ruby_version%>
19
-
20
- [Aspera APIs](https://developer.ibm.com/?size=30&q=aspera&DWContentType[0]=APIs)
21
-
22
- ## <a id="when_to_use"></a>When to use and when not to use
23
-
24
- <%=tool%> is designed to be used as a command line tool to:
25
-
26
- * execute commands on Aspera products
27
- * transfer to/from Aspera products
28
-
29
- So it is designed for:
30
-
31
- * Interactive operations on a text terminal (typically, VT100 compatible)
32
- * Batch operations in (shell) scripts (e.g. cron job)
33
-
34
- <%=tool%> can be seen as a command line tool integrating:
35
-
36
- * a configuration file (config.yaml)
37
- * advanced command line options
38
- * cURL (for REST calls)
39
- * Aspera transfer (ascp)
40
-
41
- One might be tempted to use it as an integration element, e.g. by building a command line programmatically, and then executing it. It is generally not a good idea.
42
- For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use [Aspera APIs](https://ibm.biz/aspera_api):
43
-
44
- * Product APIs (REST) : e.g. AoC, Faspex, node
45
- * Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)
46
-
47
- Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.
48
-
49
- For scripting and ad'hoc command line operations, <%=tool%> is perfect.
50
-
51
- ## <a id="parsing"></a>Notations, Shell and Command line parsing
52
-
53
- In examples, command line operations are shown using a shell such: `bash` or `zsh`.
54
-
55
- Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
56
-
57
- <%=tool%> is typically executed in a shell, either interactively or in a script. <%=tool%> receives its arguments from this shell.
58
-
59
- On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before <%=tool%> (Ruby) is executed, e.g. [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). Ruby receives a list parameters and gives it to <%=tool%>. So special character handling (quotes, spaces, env vars, ...) is done in the shell.
60
-
61
- On Windows, `cmd.exe` is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.
62
-
63
- * [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY)
64
- * [Understand Quoting and Escaping of Windows Command Line Arguments](http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/)
65
-
66
- In case of doubt of argument values after parsing test like this:
67
-
68
- ```bash
69
- <%=cmd%> conf echo "Hello World" arg2 3
70
- ```
71
-
72
- ```bash
73
- "Hello World"
74
- ERROR: Argument: unprocessed values: ["arg2", "3"]
75
- ```
76
-
77
- `echo` displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (<%=tool%>), next command line arguments are shown in the error message.
78
-
79
- ## Quick Start
80
-
81
- This section guides you from installation, first use and advanced use.
82
-
83
- First, follow the section: [Installation](#installation) (Ruby, Gem, FASP) to start using <%=tool%>.
84
-
85
- Once the gem is installed, <%=tool%> shall be accessible:
86
-
87
- ```bash
88
- <%=cmd%> --version
89
- ```
90
-
91
- ```bash
92
- <%=gemspec.version.to_s%>
93
- ```
94
-
95
- ### First use
96
-
97
- Once installation is completed, you can proceed to the first use with a demo server:
98
-
99
- If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
100
-
101
- To test with Aspera demo transfer server, setup the environment and then test:
102
-
103
- ```bash
104
- <%=cmd%> config initdemo
105
- ```
106
-
107
- ```bash
108
- <%=cmd%> server browse /
109
- ```
110
-
111
- ```output
112
- :............:...........:......:........:...........................:.......................:
113
- : zmode : zuid : zgid : size : mtime : name :
114
- :............:...........:......:........:...........................:.......................:
115
- : dr-xr-xr-x : asperaweb : fasp : 4096 : 2014-04-10 19:44:05 +0200 : aspera-test-dir-tiny :
116
- : drwxr-xr-x : asperaweb : fasp : 176128 : 2018-03-15 12:20:10 +0100 : Upload :
117
- : dr-xr-xr-x : asperaweb : fasp : 4096 : 2015-04-01 00:37:22 +0200 : aspera-test-dir-small :
118
- : dr-xr-xr-x : asperaweb : fasp : 4096 : 2018-05-04 14:26:55 +0200 : aspera-test-dir-large :
119
- :............:...........:......:........:...........................:.......................:
120
- ```
121
-
122
- If you want to use <%=tool%> with another server, and in order to make further calls more convenient, it is advised to define a <%=prst%> for the server's authentication options. The following example will:
123
-
124
- * create a <%=prst%>
125
- * define it as default for `server` plugin
126
- * list files in a folder
127
- * download a file
128
-
129
- ```bash
130
- <%=cmd%> config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_
131
- ```
132
-
133
- ```output
134
- updated: myserver
135
- ```
136
-
137
- ```bash
138
- <%=cmd%> config preset set default server myserver
139
- ```
140
-
141
- ```output
142
- updated: default&rarr;server to myserver
143
- ```
144
-
145
- ```bash
146
- <%=cmd%> server browse /aspera-test-dir-large
147
- ```
148
-
149
- ```output
150
- :............:...........:......:..............:...........................:............................:
151
- : zmode : zuid : zgid : size : mtime : name :
152
- :............:...........:......:..............:...........................:............................:
153
- : -rw-rw-rw- : asperaweb : fasp : 10133504 : 2018-05-04 14:16:24 +0200 : ctl_female_2.fastq.partial :
154
- : -rw-r--r-- : asperaweb : fasp : 209715200 : 2014-04-10 19:49:27 +0200 : 200MB :
155
- : -rw-r--r-- : asperaweb : fasp : 524288000 : 2014-04-10 19:44:15 +0200 : 500MB :
156
- : -rw-r--r-- : asperaweb : fasp : 5368709120 : 2014-04-10 19:45:52 +0200 : 5GB :
157
- : -rw-r--r-- : asperaweb : fasp : 500000000000 : 2017-06-14 20:09:57 +0200 : 500GB :
158
- : -rw-rw-rw- : asperaweb : fasp : 13606912 : 2018-05-04 14:20:21 +0200 : ctl_male_2.fastq.partial :
159
- : -rw-rw-rw- : asperaweb : fasp : 76 : 2018-05-04 14:13:18 +0200 : ctl_female_2.fastq.haspx :
160
- : -rw-rw-rw- : asperaweb : fasp : 647348 : 2018-05-04 14:26:39 +0200 : ctl_female_2.gz :
161
- : -rw-rw-rw- : asperaweb : fasp : 74 : 2018-05-04 14:16:00 +0200 : ctl_male_2.fastq.haspx :
162
- : -rw-r--r-- : asperaweb : fasp : 1048576000 : 2014-04-10 19:49:23 +0200 : 1GB :
163
- : -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
164
- : -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
165
- :............:...........:......:..............:...........................:............................:
166
- ```
167
-
168
- ```bash
169
- <%=cmd%> server download /aspera-test-dir-large/200MB
170
- ```
171
-
172
- ```output
173
- Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
174
- complete
175
- ```
176
-
177
- ### Going further
178
-
179
- Get familiar with configuration, options, commands : [Command Line Interface](#cli).
180
-
181
- Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : [Application Plugins](plugins)
182
-
183
- ## <a id="installation"></a>Installation
184
-
185
- It is possible to install *either* directly on the host operating system (Linux, Windows, macOS) or as a docker container.
186
-
187
- The direct installation is recommended and consists in installing:
188
-
189
- * [Ruby](#ruby) version <%=gemspec.required_ruby_version%>
190
- * [<%=gemspec.name%>](#the_gem)
191
- * [Aspera SDK (ascp)](#fasp_prot)
192
-
193
- The following sections provide information on the various installation methods.
194
-
195
- An internet connection is required for the installation. If you don't have internet for the installation, refer to section [Installation without internet access](#offline_install).
196
-
197
- ### Docker container
198
-
199
- Use this method only if you know what you do, else use the standard recommended method as described here above.
200
-
201
- This method installs a docker image that contains: Ruby, <%=tool%> and the FASP sdk.
202
-
203
- The image is: [https://hub.docker.com/r/martinlaurent/ascli](https://hub.docker.com/r/martinlaurent/ascli)
204
-
205
- Ensure that you have Docker installed.
206
-
207
- ```bash
208
- docker --version
209
- ```
210
-
211
- Download the wrapping script:
212
-
213
- ```bash
214
- curl -o <%=cmd%> https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli
215
- ```
216
-
217
- ```bash
218
- chmod a+x <%=cmd%>
219
- ```
220
-
221
- Install the container image:
222
-
223
- ```bash
224
- ./<%=cmd%> install
225
- ```
226
-
227
- Start using it !
228
-
229
- Note that the tool is run in the container, so transfers are also executed in the container, not calling host.
230
-
231
- The wrapping script maps the container folder `/usr/src/app/config` to configuration folder `$HOME/.aspera/<%=cmd%>` on host.
232
-
233
- To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped).
234
- To add local storage as a volume edit the script: <%=tool%> and add a `--volume` stanza.
235
-
236
- ### <a id="ruby"></a>Ruby
237
-
238
- Use this method to install on the native host.
239
-
240
- A ruby interpreter is required to run the tool or to use the gem and tool.
241
-
242
- Required Ruby version: <%=gemspec.required_ruby_version%>. Ruby version 3 is also supported.
243
-
244
- *Ruby can be installed using any method* : rpm, yum, dnf, rvm, brew, windows installer, ... .
245
-
246
- Refer to the following sections for a proposed method for specific operating systems.
247
-
248
- The recommended installation method is `rvm` for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...).
249
- If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method.
250
- If you have a simpler better way to install Ruby version <%=gemspec.required_ruby_version%> : use it !
251
-
252
- #### Generic: RVM: single user installation (not root)
253
-
254
- Use this method which provides more flexibility.
255
-
256
- Install "rvm": follow [https://rvm.io/](https://rvm.io/) :
257
-
258
- Install the 2 keys
259
-
260
- ```bash
261
- gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
262
- ```
263
-
264
- Execute the shell/curl command. As regular user, it install in the user's home: `~/.rvm` .
265
-
266
- ```bash
267
- \curl -sSL https://get.rvm.io | bash -s stable
268
- ```
269
-
270
- If you keep the same terminal (not needed if re-login):
271
-
272
- ```bash
273
- source ~/.rvm/scripts/rvm
274
- ```
275
-
276
- It is advised to get one of the pre-compiled ruby version, you can list with:
277
-
278
- ```bash
279
- rvm list --remote
280
- ```
281
-
282
- Install the chosen pre-compiled Ruby version:
283
-
284
- ```bash
285
- rvm install 2.7.2 --binary
286
- ```
287
-
288
- Ruby is now installed for the user, go on to Gem installation.
289
-
290
- #### Generic: RVM: global installation (as root)
291
-
292
- Follow the same method as single user install, but execute as "root".
293
-
294
- As root, it installs by default in /usr/local/rvm for all users and creates `/etc/profile.d/rvm.sh`.
295
- One can install in another location with :
296
-
297
- ```bash
298
- curl -sSL https://get.rvm.io | bash -s -- --path /usr/local
299
- ```
300
-
301
- As root, make sure this will not collide with other application using Ruby (e.g. Faspex).
302
- If so, one can rename the login script: `mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok`.
303
- To activate ruby (and <%=cmd%>) later, source it:
304
-
305
- ```bash
306
- source /etc/profile.d/rvm.sh.ok
307
- ```
308
-
309
- ```bash
310
- rvm version
311
- ```
312
-
313
- #### Windows: Installer
314
-
315
- Install Latest stable Ruby using [https://rubyinstaller.org/](https://rubyinstaller.org/) :
316
-
317
- * Go to "Downloads".
318
- * Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
319
- * At the end of the installer uncheck the box to skip the installation of "MSys2": not needed.
320
-
321
- #### macOS: pre-installed or `brew`
322
-
323
- macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install <%=gemspec.name%> using `sudo` :
324
-
325
- ```bash
326
- sudo gem install <%=gemspec.name%><%=geminstadd%>
327
- ```
328
-
329
- Alternatively, if you use [Homebrew](https://brew.sh/) already you can install Ruby with it:
330
-
331
- ```bash
332
- brew install ruby
333
- ```
334
-
335
- #### Linux: package
336
-
337
- If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).
338
-
339
- Example:
340
-
341
- ```bash
342
- yum install -y ruby rubygems ruby-json
343
- ```
344
-
345
- One can cleanup the whole yum-installed ruby environment like this to uninstall:
346
-
347
- ```bash
348
- gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)
349
- ```
350
-
351
- ```bash
352
- yum remove -y ruby ruby-libs
353
- ```
354
-
355
- #### Other Unixes (AIX)
356
-
357
- Ruby is sometimes made available as installable package through third party providers.
358
- For example for AIX, one can look at:
359
-
360
- <https://www.ibm.com/support/pages/aix-toolbox-open-source-software-downloads-alpha#R>
361
-
362
- If your Unix does not provide a pre-built ruby, you can get it using one of those
363
- [methods](https://www.ruby-lang.org/en/documentation/installation/).
364
-
365
- For instance to build from source, and install in `/opt/ruby` :
366
-
367
- ```bash
368
- wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
369
-
370
- gzip -d ruby-2.7.2.tar.gz
371
-
372
- tar xvf ruby-2.7.2.tar
373
-
374
- cd ruby-2.7.2
375
-
376
- ./configure --prefix=/opt/ruby
377
-
378
- make ruby.imp
379
-
380
- make
381
-
382
- make install
383
- ```
384
-
385
- If you already have a Java JVM on your system (`java`), it is possible to use `jruby`:
386
-
387
- <https://www.jruby.org/download>
388
-
389
- Note that using jruby the startup time is longer than the native ruby, but transfer speed is not impacted (executed by `ascp` binary).
390
-
391
- ### <a id="the_gem"></a>`<%=gemspec.name%>` gem
392
-
393
- Once you have Ruby and rights to install gems: Install the gem and its dependencies:
394
-
395
- ```bash
396
- gem install <%=gemspec.name%><%=geminstadd%>
397
- ```
398
-
399
- To upgrade to the latest version:
400
-
401
- ```bash
402
- gem update <%=gemspec.name%>
403
- ```
404
-
405
- <%=tool%> checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option `version_check_days` to `0`, or specify a different period in days.
406
-
407
- To check manually:
408
-
409
- ```bash
410
- <%=cmd%> conf check_update
411
- ```
412
-
413
- ### <a id="fasp_prot"></a>FASP Protocol
414
-
415
- Most file transfers will be done using the FASP protocol, using `ascp`.
416
- Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:
417
-
418
- * ascp
419
- * aspera-license (in same folder, or ../etc)
420
-
421
- This can be installed either be installing an Aspera transfer software, or using an embedded command:
422
-
423
- ```bash
424
- <%=cmd%> conf ascp install
425
- ```
426
-
427
- If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:
428
-
429
- ```bash
430
- curl -Lso SDK.zip https://ibm.biz/aspera_sdk
431
- ```
432
-
433
- ```bash
434
- <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
435
- ```
436
-
437
- The format is: `file:///<path>`, where `<path>` can be either a relative path (not starting with `/`), or an absolute path.
438
-
439
- If the embedded method is not used, the following packages are also suitable:
440
-
441
- * IBM Aspera Connect Client (Free)
442
- * IBM Aspera Desktop Client (Free)
443
- * IBM Aspera CLI (Free)
444
- * IBM Aspera High Speed Transfer Server (Licensed)
445
- * IBM Aspera High Speed Transfer EndPoint (Licensed)
446
-
447
- For instance, Aspera Connect Client can be installed
448
- by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/).
449
-
450
- <%=tool%> will detect most of Aspera transfer products in standard locations and use the first one found.
451
- Refer to section [FASP](#client) for details on how to select a client or set path to the FASP protocol.
452
-
453
- Several methods are provided to start a transfer.
454
- Use of a local client ([`direct`](#agt_direct) transfer agent) is one of them, but other methods are available. Refer to section: [Transfer Agents](#agents)
455
-
456
- ### <a id="offline_install"></a>Installation in air gapped environment
457
-
458
- Note that currently no pre-packaged version exist yet.
459
- A method to build one is provided here:
460
-
461
- The procedure:
462
-
463
- * Follow the non-root installation procedure with RVM, including gem
464
- * Archive (zip, tar) the main RVM folder (includes <%=cmd%>):
465
-
466
- ```bash
467
- cd $HOME && tar zcvf rvm-<%=cmd%>.tgz .rvm
468
- ```
469
-
470
- * Get the Aspera SDK.
471
-
472
- ```bash
473
- <%=cmd%> conf --show-config --fields=sdk_url
474
- ```
475
-
476
- * Download the SDK archive from that URL.
477
-
478
- ```bash
479
- curl -Lso SDK.zip https://ibm.biz/aspera_sdk
480
- ```
481
-
482
- * Transfer those 2 files to the target system
483
-
484
- * On target system
485
-
486
- ```bash
487
- cd $HOME
488
-
489
- tar zxvf rvm-<%=cmd%>.tgz
490
-
491
- source ~/.rvm/scripts/rvm
492
-
493
- <%=cmd%> conf ascp install --sdk-url=file:///SDK.zip
494
- ```
495
-
496
- * Add those lines to shell init (`.profile`)
497
-
498
- ```bash
499
- source ~/.rvm/scripts/rvm
500
- ```
501
-
502
- ## <a id="cli"></a>Command Line Interface: <%=tool%>
503
-
504
- The `<%=gemspec.name%>` Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):
505
-
506
- * IBM Aspera High Speed Transfer Server (FASP and Node)
507
- * IBM Aspera on Cloud (including ATS)
508
- * IBM Aspera Faspex
509
- * IBM Aspera Shares
510
- * IBM Aspera Console
511
- * IBM Aspera Orchestrator
512
- * and more...
513
-
514
- <%=tool%> provides the following features:
515
-
516
- * Supports most Aspera server products (on-premise and SaaS)
517
- * Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
518
- * Supports Commands, Option values and Parameters shortcuts
519
- * FASP [Transfer Agents](#agents) can be: local ascp, or Connect Client, or any transfer node
520
- * Transfer parameters can be altered by modification of _transfer-spec_, this includes requiring multi-session
521
- * Allows transfers from products to products, essentially at node level (using the node transfer agent)
522
- * Supports FaspStream creation (using Node API)
523
- * Supports Watchfolder creation (using Node API)
524
- * Additional command plugins can be written by the user
525
- * Supports download of faspex and Aspera on Cloud "external" links
526
- * Supports "legacy" ssh based FASP transfers and remote commands (ascmd)
527
-
528
- Basic usage is displayed by executing:
529
-
530
- ```bash
531
- <%=cmd%> -h
532
- ```
533
-
534
- Refer to sections: [Usage](#usage) and [Sample Commands](#commands).
535
-
536
- Not all <%=tool%> features are fully documented here, the user may explore commands on the command line.
537
-
538
- ### Arguments : Commands and options
539
-
540
- Arguments are the units of command line, as parsed by the shell, typically separated by spaces (and called "argv").
541
-
542
- There are two types of command line arguments: Commands and Options. Example :
543
-
544
- ```bash
545
- <%=cmd%> command subcommand --option-name=VAL1 VAL2
546
- ```
547
-
548
- * executes _command_: `command subcommand`
549
- * with one _option_: `option_name`
550
- * this option is given a _value_ of: `VAL1`
551
- * the command has one additional _argument_: `VAL2`
552
-
553
- When the value of a command, option or argument is constrained by a fixed list of values, it is possible to use the first letters of the value only, provided that it uniquely identifies a value. For example `<%=cmd%> conf ov` is the same as `<%=cmd%> config overview`.
554
-
555
- The value of options and arguments is evaluated with the [Extended Value Syntax](#extended).
556
-
557
- #### Options
558
-
559
- All options, e.g. `--log-level=debug`, are command line arguments that:
560
-
561
- * start with `--`
562
- * have a name, in lowercase, using `-` as word separator in name (e.g. `--log-level=debug`)
563
- * have a value, separated from name with a `=`
564
- * can be used by prefix, provided that it is unique. E.g. `--log-l=debug` is the same as `--log-level=debug`
565
-
566
- Exceptions:
567
-
568
- * some options accept a short form, e.g. `-Ptoto` is equivalent to `--preset=toto`, refer to the manual or `-h`.
569
- * some options (flags) don't take a value, e.g. `-r`
570
- * the special option `--` stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a `-`. Example:
571
-
572
- ```bash
573
- <%=cmd%> config echo -- --sample
574
- ```
575
-
576
- ```bash
577
- "--sample"
578
- ```
579
-
580
- Note that here, `--sample` is taken as an argument, and not as an option, due to `--`.
581
-
582
- Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.
583
-
584
- The value for _any_ options can come from the following locations (in this order, last value evaluated overrides previous value):
585
-
586
- * [Configuration file](#configfile).
587
- * Environment variable
588
- * Command line
589
-
590
- Environment variable starting with prefix: <%=evp%> are taken as option values, e.g. `<%=evp%>OPTION_NAME` is for `--option-name`.
591
-
592
- Options values can be displayed for a given command by providing the `--show-config` option: `<%=cmd%> node --show-config`
593
-
594
- #### Commands and Arguments
595
-
596
- Command line arguments that are not options are either commands or arguments. If an argument must begin with `-`, then either use the `@val:` syntax (see [Extended Values](#extended)), or use the `--` separator (see above).
597
-
598
- ### Interactive Input
599
-
600
- Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.
601
-
602
- The behavior can be controlled with:
603
-
604
- * --interactive=&lt;yes|no&gt; (default=yes if STDIN is a terminal, else no)
605
- * yes : missing mandatory parameters/options are asked to the user
606
- * no : missing mandatory parameters/options raise an error message
607
- * --ask-options=&lt;yes|no&gt; (default=no)
608
- * optional parameters/options are asked to user
609
-
610
- ### Output
611
-
612
- Command execution will result in output (terminal, stdout/stderr).
613
- The information displayed depends on the action.
614
-
615
- #### Types of output data
616
-
617
- Depending on action, the output will contain:
618
-
619
- * `single_object` : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
620
- * `object_list` : displayed as a 2 dimensional table: one line per item, one column per attribute.
621
- * `value_list` : a table with one column.
622
- * `empty` : nothing
623
- * `status` : a message
624
- * `other_struct` : a complex structure that cannot be displayed as an array
625
-
626
- #### Format of output
627
-
628
- By default, result of type single_object and object_list are displayed using format `table`.
629
- The table style can be customized with parameter: `table_style` (horizontal, vertical and intersection characters) and is `:.:` by default.
630
-
631
- In a table format, when displaying "objects" (single, or list), by default, sub object are
632
- flattened (option `flat_hash`). So, object {"user":{"id":1,"name":"toto"}} will have attributes: user.id and user.name.
633
- Setting `flat_hash` to `false` will only display one field: "user" and value is the sub hash table.
634
- When in flatten mode, it is possible to filter fields by "dotted" field name.
635
-
636
- Object lists are displayed one per line, with attributes as columns. Single objects are transposed: one attribute per line.
637
- If transposition of single object is not desired, use option: `transpose_single` set to `no`.
638
-
639
- The style of output can be set using the `format` parameter, supporting:
640
-
641
- * `table` : Text table
642
- * `ruby` : Ruby code
643
- * `json` : JSON code
644
- * `jsonpp` : JSON pretty printed
645
- * `yaml` : YAML
646
- * `csv` : Comma Separated Values
647
-
648
- #### <a id="option_select"></a>Option: `select`: Filter on columns values for `object_list`
649
-
650
- Table output can be filtered using the `select` parameter. Example:
651
-
652
- ```javascript
653
- <%=cmd%> aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
654
- ```
655
-
656
- ```output
657
- :...............................:..................................:...........:
658
- : name : email : ats_admin :
659
- :...............................:..................................:...........:
660
- : John Curtis : john@example.com : true :
661
- : Laurent Martin : laurent@example.com : true :
662
- :...............................:..................................:...........:
663
- ```
664
-
665
- Note that `select` filters selected elements from the result of API calls, while the `query` parameters gives filtering parameters to the API when listing elements.
666
-
667
- #### Verbosity of output
668
-
669
- Output messages are categorized in 3 types:
670
-
671
- * `info` output contain additional information, such as number of elements in a table
672
- * `data` output contain the actual output of the command (object, or list of objects)
673
- * `error`output contain error messages
674
-
675
- The option `display` controls the level of output:
676
-
677
- * `info` displays all messages
678
- * `data` display `data` and `error` messages
679
- * `error` display only error messages.
680
-
681
- #### Selection of output object properties
682
-
683
- By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the `fields` option to one of the following value:
684
-
685
- * DEF : default display of columns (that's the default, when not set)
686
- * ALL : all columns available
687
- * a,b,c : the list of attributes specified by the comma separated list
688
- * Array extended value: for instance, @json:'["a","b","c"]' same as above
689
- * +a,b,c : add selected properties to the default selection.
690
- * -a,b,c : remove selected properties from the default selection.
691
-
692
- ### <a id="extended"></a>Extended Value Syntax
693
-
694
- Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
695
-
696
- The extended value syntax is:
697
-
698
- ```bash
699
- <0 or more decoders><0 or 1 reader><nothing or some text value>
700
- ```
701
-
702
- The difference between reader and decoder is order and ordinality. Both act like a function of value on right hand side. Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
703
-
704
- The following "readers" are supported (returns value in []):
705
-
706
- * @val:VALUE : [String] prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
707
- * @file:PATH : [String] read value from a file (prefix `~/` is replaced with the users home folder), e.g. `--key=@file:~/.ssh/mykey`
708
- * @path:PATH : [String] performs path expansion (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml`
709
- * @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
710
- * @stdin: : [String] read from stdin (no value on right)
711
- * @preset:NAME : [Hash] get whole <%=opprst%> value by name. Subvalues can also be used using `.` as separator. e.g. foo.bar is conf[foo][bar]
712
-
713
- In addition it is possible to decode a value, using one or multiple decoders :
714
-
715
- * @base64: [String] decode a base64 encoded string
716
- * @json: [any] decode JSON values (convenient to provide complex structures)
717
- * @zlib: [String] uncompress data
718
- * @ruby: [any] execute ruby code
719
- * @csvt: [Array] decode a titled CSV value
720
- * @lines: [Array] split a string in multiple lines and return an array
721
- * @list: [Array] split a string in multiple items taking first character as separator and return an array
722
- * @incps: [Hash] include values of presets specified by key `incps` in input hash
723
-
724
- To display the result of an extended value, use the `config echo` command.
725
-
726
- Example: read the content of the specified file, then, base64 decode, then unzip:
727
-
728
- ```bash
729
- <%=cmd%> config echo @zlib:@base64:@file:myfile.dat
730
- ```
731
-
732
- Example: create a value as a hash, with one key and the value is read from a file:
733
-
734
- ```bash
735
- <%=cmd%> config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'
736
- ```
737
-
738
- Example: read a csv file and create a list of hash for bulk provisioning:
739
-
740
- ```bash
741
- cat test.csv
742
- ```
743
-
744
- ```bash
745
- name,email
746
- lolo,laurent@example.com
747
- toto,titi@tutu.tata
748
- ```
749
-
750
- ```bash
751
- <%=cmd%> config echo @csvt:@file:test.csv
752
- ```
753
-
754
- ```output
755
- :......:.....................:
756
- : name : email :
757
- :......:.....................:
758
- : lolo : laurent@example.com :
759
- : toto : titi@tutu.tata :
760
- :......:.....................:
761
- ```
762
-
763
- Example: create a hash and include values from preset named "config" of config file in this hash
764
-
765
- ```javascript
766
- <%=cmd%> config echo @incps:@json:'{"hello":true,"incps":["config"]}'
767
- ```
768
-
769
- ```bash
770
- {"version"=>"0.9", "hello"=>true}
771
- ```
772
-
773
- Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
774
-
775
- ### <a id="native"></a>Structured Value
776
-
777
- Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
778
-
779
- For instance, a [_transfer-spec_](#transferspec) is expected to be a _Structured Value_.
780
-
781
- Structured values shall be described using the [Extended Value Syntax](#extended).
782
- A convenient way to specify a _Structured Value_ is to use the `@json:` decoder, and describe the value in JSON format. The `@ruby:` decoder can also be used. For an array of hash tables, the `@csvt:` decoder can be used.
783
-
784
- It is also possible to provide a _Structured Value_ in a file using `@json:@file:<path>`
785
-
786
- ### <a id="conffolder"></a>Configuration and Persistency Folder
787
-
788
- <%=tool%> configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored in folder `[User's home folder]/.aspera/<%=cmd%>`.
789
-
790
- Note: `[User's home folder]` is found using ruby's `Dir.home` (`rb_w32_home_dir`).
791
- It uses the `HOME` env var primarily, and on MS Windows it also looks at `%HOMEDRIVE%%HOMEPATH%` and `%USERPROFILE%`. <%=tool%> sets the env var `%HOME%` to the value of `%USERPROFILE%` if set and exists. So, on Windows `%USERPROFILE%` is used as it is more reliable than `%HOMEDRIVE%%HOMEPATH%`.
792
-
793
- The main folder can be displayed using :
794
-
795
- ```bash
796
- <%=cmd%> config folder
797
- ```
798
-
799
- ```bash
800
- /Users/kenji/.aspera/<%=cmd%>
801
- ```
802
-
803
- It can be overridden using the environment variable `<%=evp%>HOME`.
804
-
805
- Example (Windows):
806
-
807
- ```output
808
- set <%=evp%>HOME=C:\Users\Kenji\.aspera\<%=cmd%>
809
-
810
- <%=cmd%> config folder
811
-
812
- C:\Users\Kenji\.aspera\<%=cmd%>
813
- ```
814
-
815
- ### <a id="configfile"></a>Configuration file
816
-
817
- On the first execution of <%=tool%>, an empty configuration file is created in the configuration folder.
818
- Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.
819
-
820
- Although the file is a standard YAML file, <%=tool%> provides commands to read and modify it using the `config` command.
821
-
822
- All options for <%=tool%> can be set on command line, or by env vars, or using <%=prsts%> in the configuration file.
823
-
824
- A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.
825
-
826
- The default configuration file is: `$HOME/.aspera/<%=cmd%>/config.yaml` (this can be overridden with option `--config-file=path` or equivalent env var).
827
-
828
- The configuration file is simply a catalog of pre-defined lists of options, called: <%=prsts%>. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a <%=prst%> (e.g. `mypreset`) using the option: `-Pmypreset` or `--preset=mypreset`.
829
-
830
- #### <a id="lprt"></a><%=prstt%>
831
-
832
- A <%=prst%> is simply a collection of parameters and their associated values in a named section in the configuration file.
833
-
834
- A named <%=prst%> can be modified directly using <%=tool%>, which will update the configuration file :
835
-
836
- ```bash
837
- <%=cmd%> config preset set|delete|show|initialize|update <<%=opprst%>>
838
- ```
839
-
840
- The command `update` allows the easy creation of <%=prst%> by simply providing the options in their command line format, e.g. :
841
-
842
- ```bash
843
- <%=cmd%> config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_demo_pass_ --ts=@json:'{"precalculate_job_size":true}'
844
- ```
845
-
846
- * This creates a <%=prst%> `demo_server` with all provided options.
847
-
848
- The command `set` allows setting individual options in a <%=prst%>.
849
-
850
- ```bash
851
- <%=cmd%> config preset set demo_server password _demo_pass_
852
- ```
853
-
854
- The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a _[Structured Value](#native)_.
855
-
856
- ```javascript
857
- <%=cmd%> config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_demo_pass_","ts":{"precalculate_job_size":true}}'
858
- ```
859
-
860
- A full terminal based overview of the configuration can be displayed using:
861
-
862
- ```bash
863
- <%=cmd%> config preset over
864
- ```
865
-
866
- A list of <%=prst%> can be displayed using:
867
-
868
- ```bash
869
- <%=cmd%> config preset list
870
- ```
871
-
872
- A good practice is to not manually edit the configuration file and use modification commands instead.
873
- If necessary, the configuration file can opened in a text editor with:
874
-
875
- ```bash
876
- <%=cmd%> config open
877
- ```
878
-
879
- Older format for commands are still supported:
880
-
881
- ```bash
882
- <%=cmd%> config id <name> set|delete|show|initialize|update
883
- <%=cmd%> config over
884
- <%=cmd%> config list
885
- ```
886
-
887
-
888
- #### <a id="lprtconf"></a>Special <%=prstt%>: config
889
-
890
- This preset name is reserved and contains a single key: `version`. This is the version of <%=tool%> which created the file.
891
-
892
- #### <a id="lprtdef"></a>Special <%=prstt%>: default
893
-
894
- This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.
895
-
896
- When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option --no-default (or -N) is used.
897
-
898
- Note that special plugin name: `config` can be associated with a preset that is loaded initially, typically used for default values.
899
-
900
- Operations on this preset are done using regular `config` operations:
901
-
902
- ```bash
903
- <%=cmd%> config preset set default _plugin_name_ _default_preset_for_plugin_
904
- ```
905
-
906
- ```bash
907
- <%=cmd%> config preset get default _plugin_name_
908
- ```
909
-
910
- ```javascript
911
- "_default_preset_for_plugin_"
912
- ```
913
-
914
- #### <a id="lplugconf"></a>Special Plugin: config
915
-
916
- Plugin `config` (not to be confused with <%=prstt%> config) is used to configure <%=tool%> but it also contains global options.
917
-
918
- When <%=tool%> starts, it looks for the `default` <%=prstt%> and if there is a value for `config`, if so, it loads the option values for any plugin used.
919
-
920
- If no global default is set by the user, the tool will use `global_common_defaults` when setting global parameters (e.g. `conf ascp use`)
921
-
922
- #### Format of file
923
-
924
- The configuration file is a hash in a YAML file. Example:
925
-
926
- ```yaml
927
- config:
928
- version: 0.3.7
929
- default:
930
- config: cli_default
931
- server: demo_server
932
- cli_default:
933
- interactive: no
934
- demo_server:
935
- url: ssh://demo.asperasoft.com:33001
936
- username: asperaweb
937
- password: _demo_pass_
938
- ```
939
-
940
- We can see here:
941
-
942
- * The configuration was created with CLI version 0.3.7
943
- * the default <%=prst%> to load for `server` plugin is : `demo_server`
944
- * the <%=prst%> `demo_server` defines some parameters: the URL and credentials
945
- * the default <%=prst%> to load in any case is : `cli_default`
946
-
947
- Two <%=prsts%> are reserved:
948
-
949
- * `config` contains a single value: `version` showing the CLI
950
- version used to create the configuration file. It is used to check compatibility.
951
- * `default` is reserved to define the default <%=prst%> name used for known plugins.
952
-
953
- The user may create as many <%=prsts%> as needed. For instance, a particular <%=prst%> can be created for a particular application instance and contain URL and credentials.
954
-
955
- Values in the configuration also follow the [Extended Value Syntax](#extended).
956
-
957
- Note: if the user wants to use the [Extended Value Syntax](#extended) inside the configuration file, using the `config preset update` command, the user shall use the `@val:` prefix. Example:
958
-
959
- ```bash
960
- <%=cmd%> config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/<%=cmd%>/aocapikey"
961
- ```
962
-
963
- This creates the <%=prst%>:
964
-
965
- ```yaml
966
- ...
967
- my_aoc_org:
968
- private_key: @file:"/Users/laurent/.aspera/<%=cmd%>/aocapikey"
969
- ...
970
- ```
971
-
972
- So, the key file will be read only at execution time, but not be embedded in the configuration file.
973
-
974
- #### Options evaluation order
975
-
976
- Some options are global, some options are available only for some plugins. (the plugin is the first level command).
977
-
978
- Options are loaded using this algorithm:
979
-
980
- * If option `--no-default` (or `-N`) is specified, then no default value is loaded is loaded for the plugin
981
- * else it looks for the name of the plugin as key in section `default`, the value is the name of the default <%=prst%> for it, and loads it.
982
- * If option `--preset=<name or extended value hash>` is specified (or `-Pxxxx`), this reads the <%=prst%> specified from the configuration file, or of the value is a Hash, it uses it as options values.
983
- * Environment variables are evaluated
984
- * Command line options are evaluated
985
-
986
- Parameters are evaluated in the order of command line.
987
-
988
- To avoid loading the default <%=prst%> for a plugin, use: `-N`
989
-
990
- On command line, words in parameter names are separated by a dash, in configuration file, separator
991
- is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.
992
-
993
- The main plugin name is `config`, so it is possible to define a default <%=prst%> for the main plugin with:
994
-
995
- ```bash
996
- <%=cmd%> config preset set cli_default interactive no
997
- ```
998
-
999
- ```bash
1000
- <%=cmd%> config preset set default config cli_default
1001
- ```
1002
-
1003
- A <%=prst%> value can be removed with `unset`:
1004
-
1005
- ```bash
1006
- <%=cmd%> config preset unset cli_default interactive
1007
- ```
1008
-
1009
- Example: Define options using command line:
1010
-
1011
- ```bash
1012
- <%=cmd%> -N --url=x --password=y --username=y node --show-config
1013
- ```
1014
-
1015
- Example: Define options using a hash:
1016
-
1017
- ```javascript
1018
- <%=cmd%> -N --preset=@json:'{"url":"x","password":"y","username":"y"}' node --show-config
1019
- ```
1020
-
1021
- #### Shares Examples
1022
-
1023
- For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console,
1024
- only username/password and url are required (either on command line, or from config file).
1025
- Those can usually be provided on the command line:
1026
-
1027
- ```bash
1028
- <%=cmd%> shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra
1029
- ```
1030
-
1031
- This can also be provisioned in a config file:
1032
-
1033
- * Build <%=prst%>
1034
-
1035
- ```bash
1036
- <%=cmd%> config preset set shares06 url https://10.25.0.6
1037
- <%=cmd%> config preset set shares06 username john
1038
- <%=cmd%> config preset set shares06 password 4sp3ra
1039
- ```
1040
-
1041
- This can also be done with one single command:
1042
-
1043
- ```javascript
1044
- <%=cmd%> config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'
1045
- ```
1046
-
1047
- or
1048
-
1049
- ```bash
1050
- <%=cmd%> config preset update shares06 --url=https://10.25.0.6 --username=john --password=4sp3ra
1051
- ```
1052
-
1053
- * Define this <%=prst%> as the default <%=prst%> for the specified plugin (`shares`)
1054
-
1055
- ```bash
1056
- <%=cmd%> config preset set default shares shares06
1057
- ```
1058
-
1059
- * Display the content of configuration file in table format
1060
-
1061
- ```bash
1062
- <%=cmd%> config overview
1063
- ```
1064
-
1065
- * Execute a command on the shares application using default parameters
1066
-
1067
- ```bash
1068
- <%=cmd%> shares repo browse /
1069
- ```
1070
-
1071
- ### <a id="vault"></a>Secret Vault
1072
-
1073
- When a secret or password is needed, it is possible to store in the secret vault.
1074
-
1075
- By default the vault is defined using option `secrets`, which can be stored in the configuration file.
1076
-
1077
- #### Using system keychain
1078
-
1079
- Only on macOS.
1080
-
1081
- It is possible to store secrets in macOS keychain (only read supported currently).
1082
-
1083
- Set option `secrets` to value `system` to use the default keychain or use value `system:[name]` to use a custom keychain.
1084
-
1085
- #### Modern config file format: encrypted in config file
1086
-
1087
- It is possible to store and use secrets encrypted.
1088
- For this use the `config vault` command.
1089
-
1090
- The vault can be initialized with `config vault init`
1091
-
1092
- Then secrets can be manipulated using commands:
1093
-
1094
- * `set`
1095
- * `get`
1096
- * `list`
1097
- * `delete`
1098
-
1099
- Secrets must be uniquely identified by `url` and `username`. An optional description can be provided using option `value`.
1100
-
1101
- #### Legacy config file format
1102
-
1103
- The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.
1104
-
1105
- For example, choose a repository name, for example `my_secrets`, and populate it like this:
1106
-
1107
- ```bash
1108
- <%=cmd%> conf id my_secrets set 'access_key1' 'secret1'
1109
-
1110
- <%=cmd%> conf id my_secrets set 'access_key2' 'secret2'
1111
-
1112
- <%=cmd%> conf id default get config
1113
-
1114
- cli_default
1115
- ```
1116
-
1117
- Here above, one has already set a `config` global preset to preset `cli_default` (refer to earlier in documentation).
1118
- So the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
1119
-
1120
- ```bash
1121
- <%=cmd%> conf id cli_default set secrets @val:@preset:my_secrets
1122
- ```
1123
-
1124
- A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
1125
-
1126
- To test if a secret can be found use:
1127
-
1128
- ```bash
1129
- <%=cmd%> conf vault get --username=access_key1
1130
- ```
1131
-
1132
- ### Plugins
1133
-
1134
- The CLI tool uses a plugin mechanism.
1135
- The first level command (just after <%=tool%> on the command line) is the name of the concerned plugin which will execute the command.
1136
- Each plugin usually represents commands sent to a specific application.
1137
- For instance, the plugin `faspex` allows operations on the application "Aspera Faspex".
1138
-
1139
- Available plugins can be found using command:
1140
-
1141
- ```bash
1142
- <%=cmd%> conf plugin list
1143
- ```
1144
-
1145
- ```output
1146
- +--------------+-----------------------------------------------------------------------------------+
1147
- | plugin | path |
1148
- +--------------+-----------------------------------------------------------------------------------+
1149
- | shares | /Users/laurent/workspace/aspera/aspera-cli/lib/aspera/cli/plugins/shares.rb |
1150
- | node | /Users/laurent/workspace/aspera/aspera-cli/lib/aspera/cli/plugins/node.rb |
1151
- ...
1152
- +--------------+-----------------------------------------------------------------------------------+
1153
- ```
1154
-
1155
- #### <a id="createownplugin"></a>Create your own plugin
1156
-
1157
- By default plugins are looked-up in folders specifed by (multi-value) option `plugin_folder`:
1158
-
1159
- ```javascript
1160
- ascli --show-config --select=@json:'{"key":"plugin_folder"}'
1161
- ```
1162
-
1163
- You can create the skeleton of a new plugin like this:
1164
-
1165
- ```bash
1166
- <%=cmd%> conf plugin create foo .
1167
- ```
1168
-
1169
- ```output
1170
- Created ./foo.rb
1171
- ```
1172
-
1173
- ```bash
1174
- <%=cmd%> --plugin-folder=. foo
1175
- ```
1176
-
1177
- #### <a id="plugins"></a>Plugins: Application URL and Authentication
1178
-
1179
- <%=tool%> comes with several Aspera application plugins.
1180
-
1181
- REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.
1182
-
1183
- Those are using options:
1184
-
1185
- * url
1186
- * username
1187
- * password
1188
-
1189
- Those can be provided using command line, parameter set, env var, see section above.
1190
-
1191
- Aspera on Cloud relies on Oauth, refer to the [Aspera on Cloud](#aoc) section.
1192
-
1193
- ### Logging, Debugging
1194
-
1195
- The gem is equipped with traces. By default logging level is `warn`.
1196
- To increase debug level, use parameter `log_level` (e.g. using command line `--log-level=xx`, env var `<%=evp%>LOG_LEVEL`, or parameter in con file).
1197
-
1198
- It is also possible to activate traces before initialization using env var `AS_LOG_LEVEL`.
1199
-
1200
- By default passwords and secrets are removed from logs.
1201
- Use option `log_passwords` to change this behaviour.
1202
-
1203
- ### Learning Aspera Product APIs (REST)
1204
-
1205
- This CLI uses REST APIs.
1206
- To display HTTP calls, use argument `-r` or `--rest-debug`, this is useful to display exact content of HTTP requests and responses.
1207
-
1208
- In order to get traces of execution, use argument : `--log-level=debug`
1209
-
1210
- ### <a id="http_options"></a>HTTP socket parameters
1211
-
1212
- If the server does not provide a valid certificate, use option: `--insecure=yes`.
1213
-
1214
- Ruby HTTP socket parameters can be adjusted.
1215
-
1216
- | parameter | default |
1217
- |----------------------|---------|
1218
- | `read_timeout` | 60 |
1219
- | `write_timeout` | 60 |
1220
- | `open_timeout` | 60 |
1221
- | `keep_alive_timeout` | 2 |
1222
-
1223
- Values are in set **seconds** and can be of type either integer or float.
1224
- Default values are the ones of Ruby.
1225
- For details refer to the Ruby library: [`Net::HTTP`](https://ruby-doc.org/stdlib/libdoc/net/http/rdoc/Net/HTTP.html).
1226
-
1227
- Like any other option, those can be set either on command line, or in config file, either in a global preset or server-specific one.
1228
-
1229
- Example:
1230
-
1231
- ```javascript
1232
- <%=cmd%> aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
1233
- ```
1234
-
1235
- ### <a id="graphical"></a>Graphical Interactions: Browser and Text Editor
1236
-
1237
- Some actions may require the use of a graphical tool:
1238
-
1239
- * a browser for Aspera on Cloud authentication (web auth method)
1240
- * a text editor for configuration file edition
1241
-
1242
- By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable.
1243
- It is also possible to force the graphical mode with option --ui :
1244
-
1245
- * `--ui=graphical` forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
1246
- * `--ui=text` forces a text environment, the URL or file path to open is displayed on terminal.
1247
-
1248
- ### HTTP proxy for REST
1249
-
1250
- To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honored by Ruby when calling REST APIs.
1251
-
1252
- ### <a id="certificates"></a>SSL CA certificate bundle
1253
-
1254
- <%=tool%> uses ruby `openssl` gem, which uses the `openssl` library, so certificates are checked against the ruby default certificates [OpenSSL::X509::DEFAULT_CERT_FILE](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html), which are typically the ones of `openssl` on Unix systems (Linux, macOS, etc..). The environment variables `SSL_CERT_FILE` and `SSL_CERT_DIR` are used if defined.
1255
-
1256
- `ascp` also needs to validate certificates when using WSS. By default, `ascp` uses primarily certificates from hard coded path (e.g. on macOS: `/Library/Aspera/ssl`). <%=tool%> overrides and sets the default ruby certificate path as well for `ascp` using `-i` switch. So to update certificates, update ruby's `openssl` gem, or use env vars `SSL_CERT_*`.
1257
-
1258
- ### Proxy auto config
1259
-
1260
- The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file, by its URL for local FASP agent. Supported schemes are : http:, https: and file:.
1261
-
1262
- The PAC file can be tested with command: `config proxy_check` , example:
1263
-
1264
- ```bash
1265
- <%=cmd%> config proxy_check --fpac=file:///./proxy.pac http://www.example.com
1266
- PROXY proxy.example.com:8080
1267
- ```
1268
-
1269
- This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
1270
-
1271
- ### <a id="client"></a>FASP configuration
1272
-
1273
- The `config` plugin also allows specification for the use of a local FASP client. It provides the following commands for `ascp` subcommand:
1274
-
1275
- * `show` : shows the path of ascp used
1276
- * `use` : list,download connect client versions available on internet
1277
- * `products` : list Aspera transfer products available locally
1278
- * `connect` : list,download connect client versions available on internet
1279
-
1280
- #### Show path of currently used `ascp`
1281
-
1282
- ```bash
1283
- <%=cmd%> config ascp show
1284
- ```
1285
-
1286
- ```output
1287
- /Users/laurent/.aspera/ascli/sdk/ascp
1288
- ```
1289
-
1290
- ```bash
1291
- <%=cmd%> config ascp info
1292
- ```
1293
-
1294
- ```output
1295
- +--------------------+-----------------------------------------------------------+
1296
- | key | value |
1297
- +--------------------+-----------------------------------------------------------+
1298
- | ascp | /Users/laurent/.aspera/ascli/sdk/ascp |
1299
- ...
1300
- ```
1301
-
1302
- #### Selection of `ascp` location for [`direct`](#agt_direct) agent
1303
-
1304
- By default, <%=tool%> uses any found local product with ascp, including SDK.
1305
-
1306
- To temporarily use an alternate ascp path use option `ascp_path` (`--ascp-path=`)
1307
-
1308
- For a permanent change, the command `config ascp use` sets the same parameter for the global default.
1309
-
1310
- Using a POSIX shell:
1311
-
1312
- ```bash
1313
- <%=cmd%> config ascp use @path:'~/Applications/Aspera CLI/bin/ascp'
1314
- ```
1315
-
1316
- ```output
1317
- ascp version: 4.0.0.182279
1318
- Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
1319
- Saved to default global preset global_common_defaults
1320
- ```
1321
-
1322
- Windows:
1323
-
1324
- ```bash
1325
- <%=cmd%> config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1326
- ```
1327
-
1328
- ```output
1329
- ascp version: 4.0.0.182279
1330
- Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
1331
- Saved to default global preset global_common_defaults
1332
- ```
1333
-
1334
- If the path has spaces, read section: [Shell and Command line parsing](#parsing).
1335
-
1336
- #### List locally installed Aspera Transfer products
1337
-
1338
- Locally installed Aspera products can be listed with:
1339
-
1340
- ```bash
1341
- <%=cmd%> config ascp products list
1342
- ```
1343
-
1344
- ```output
1345
- :.........................................:................................................:
1346
- : name : app_root :
1347
- :.........................................:................................................:
1348
- : Aspera Connect : /Users/laurent/Applications/Aspera Connect.app :
1349
- : IBM Aspera CLI : /Users/laurent/Applications/Aspera CLI :
1350
- : IBM Aspera High-Speed Transfer Endpoint : /Library/Aspera :
1351
- : Aspera Drive : /Applications/Aspera Drive.app :
1352
- :.........................................:................................................:
1353
- ```
1354
-
1355
- #### Selection of local client for `ascp` for [`direct`](#agt_direct) agent
1356
-
1357
- If no ascp is selected, this is equivalent to using option: `--use-product=FIRST`.
1358
-
1359
- Using the option use_product finds the ascp binary of the selected product.
1360
-
1361
- To permanently use the ascp of a product:
1362
-
1363
- ```bash
1364
- <%=cmd%> config ascp products use 'Aspera Connect'
1365
- saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp
1366
- ```
1367
-
1368
- #### Installation of Connect Client on command line
1369
-
1370
- ```bash
1371
- <%=cmd%> config ascp connect list
1372
- ```
1373
-
1374
- ```output
1375
- +-----------------------------------------------+--------------------------------------+-----------+
1376
- | id | title | version |
1377
- +-----------------------------------------------+--------------------------------------+-----------+
1378
- | urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E | Aspera Connect for Windows | 3.11.2.63 |
1379
- | urn:uuid:A3820D20-083E-11E2-892E-0800200C9A66 | Aspera Connect for Windows 64-bit | 3.11.2.63 |
1380
- | urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E | Aspera Connect for Windows XP | 3.11.2.63 |
1381
- | urn:uuid:55425020-083E-11E2-892E-0800200C9A66 | Aspera Connect for Windows XP 64-bit | 3.11.2.63 |
1382
- | urn:uuid:D8629AD2-6898-4811-A46F-2AF386531BFF | Aspera Connect for Mac Intel | 3.11.2.63 |
1383
- | urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 | Aspera Connect for Linux 64 | 3.11.2.63 |
1384
- +-----------------------------------------------+--------------------------------------+-----------+
1385
- ```
1386
-
1387
- ```bash
1388
- <%=cmd%> config ascp connect version 'Aspera Connect for Mac Intel' list
1389
- ```
1390
-
1391
- ```output
1392
- +-------------------------------------------+--------------------------+-----------------------------------------------------------------------------------------+----------+---------------------+
1393
- | title | type | href | hreflang | rel |
1394
- +-------------------------------------------+--------------------------+-----------------------------------------------------------------------------------------+----------+---------------------+
1395
- | Mac Intel Installer | application/octet-stream | bin/IBMAsperaConnectInstaller-3.11.2.63.dmg | en | enclosure |
1396
- | Mac Intel Installer | application/octet-stream | bin/IBMAsperaConnectInstallerOneClick-3.11.2.63.dmg | en | enclosure-one-click |
1397
- | Aspera Connect for Mac HTML Documentation | text/html | https://www.ibm.com/docs/en/aspera-connect/3.11?topic=aspera-connect-user-guide-macos | en | documentation |
1398
- | Aspera Connect for Mac Release Notes | text/html | https://www.ibm.com/docs/en/aspera-connect/3.11?topic=notes-release-aspera-connect-3112 | en | release-notes |
1399
- +-------------------------------------------+--------------------------+-----------------------------------------------------------------------------------------+----------+---------------------+
1400
- ```
1401
-
1402
- ```bash
1403
- <%=cmd%> config ascp connect version 'Aspera Connect for Mac Intel' download enclosure --to-folder=.
1404
- ```
1405
-
1406
- ```output
1407
- Time: 00:00:02 ======================================================================= 100% 27766 KB/sec Time: 00:00:02
1408
- Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg
1409
- ```
1410
-
1411
- ### <a id="agents"></a>Transfer Agents
1412
-
1413
- Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
1414
-
1415
- When a transfer needs to be started, a [_transfer-spec_](#transferspec) has been internally prepared.
1416
- This [_transfer-spec_](#transferspec) will be executed by a transfer client, here called "Transfer Agent".
1417
-
1418
- There are currently 3 agents:
1419
-
1420
- * [`direct`](#agt_direct) : a local execution of `ascp`
1421
- * [`connect`](#agt_connect) : use of a local Connect Client
1422
- * [`node`](#agt_node) : use of an Aspera Transfer Node (potentially _remote_).
1423
- * [`httpgw`](#agt_httpgw) : use of an Aspera HTTP Gateway
1424
- * [`trsdk`](#agt_trsdk) : use of Aspera Transfer SDK
1425
-
1426
- Note that all transfer operation are seen from the point of view of the agent.
1427
- For instance, a node agent making an "upload", or "package send" operation,
1428
- will effectively push files to the related server from the agent node.
1429
-
1430
- <%=tool%> standardizes on the use of a [_transfer-spec_](#transferspec) instead of _raw_ ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.
1431
-
1432
- #### <a id="agt_direct"></a>Direct
1433
-
1434
- The `direct` agent directly executes a local ascp.
1435
- This is the default for <%=tool%>.
1436
- This is equivalent to specifying `--transfer=direct`.
1437
- <%=tool%> will detect locally installed Aspera products, including SDK.
1438
- Refer to section [FASP](#client).
1439
-
1440
- The `transfer-info` accepts the following optional parameters to control multi-session, WSS
1441
-
1442
- <table>
1443
- <tr><th>Name</th><th>Type</th><th>Description</th></tr>
1444
- <tr><td>wss</td><td>Bool</td><td>Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: false</td></tr>
1445
- <tr><td>spawn_timeout_sec</td><td>Float</td><td>Multi session<br/>Verification time that ascp is running<br/>Default: 3</td></tr>
1446
- <tr><td>spawn_delay_sec</td><td>Float</td><td>Multi session<br/>Delay between startup of sessions<br/>Default: 2</td></tr>
1447
- <tr><td>multi_incr_udp</td><td>Bool</td><td>Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true</td></tr>
1448
- <tr><td>resume</td><td>Hash</td><td>Resume<br/>parameters<br/>See below</td></tr>
1449
- <tr><td>resume.iter_max</td><td>int</td><td>Resume<br/>Max number of retry on error<br/>Default: 7</td></tr>
1450
- <tr><td>resume.sleep_initial</td><td>int</td><td>Resume<br/>First Sleep before retry<br/>Default: 2</td></tr>
1451
- <tr><td>resume.sleep_factor</td><td>int</td><td>Resume<br/>Multiplier of sleep period between attempts<br/>Default: 2</td></tr>
1452
- <tr><td>resume.sleep_max</td><td>int</td><td>Resume<br/>Default: 60</td></tr>
1453
- </table>
1454
-
1455
- Resume: In case of transfer interruption, the agent will resume a transfer up to `iter_max` time.
1456
- Sleep between iterations is:
1457
-
1458
- ```bash
1459
- max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )
1460
- ```
1461
-
1462
- Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).
1463
-
1464
- Examples:
1465
-
1466
- ```javascript
1467
- <%=cmd%> ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
1468
- <%=cmd%> ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
1469
- ```
1470
-
1471
- To specify a FASP proxy (only supported with the `direct` agent), set the appropriate [_transfer-spec_](#transferspec) parameter:
1472
-
1473
- * `EX_fasp_proxy_url`
1474
- * `EX_http_proxy_url` (proxy for legacy http fallback)
1475
- * `EX_ascp_args`
1476
-
1477
- #### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
1478
-
1479
- By specifying option: `--transfer=connect`, <%=tool%> will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
1480
-
1481
- #### <a id="agt_node"></a>Aspera Node API : Node to node transfers
1482
-
1483
- By specifying option: `--transfer=node`, the CLI will start transfers in an Aspera
1484
- Transfer Server using the Node API, either on a local or remote node.
1485
- Parameters provided in option `transfer_info` are:
1486
-
1487
- <table>
1488
- <tr><th>Name</th><th>Type</th><th>Description</th></tr>
1489
- <tr><td>url</td><td>string</td><td>URL of the node API</br>Mandatory</td></tr>
1490
- <tr><td>username</td><td>string</td><td>node api user or access key</br>Mandatory</td></tr>
1491
- <tr><td>password</td><td>string</td><td>password, secret or bearer token</br>Mandatory</td></tr>
1492
- <tr><td>root_id</td><td>string</td><td>password or secret</br>Mandatory only for bearer token</td></tr>
1493
- </table>
1494
-
1495
- Like any other option, `transfer_info` can get its value from a pre-configured <%=prst%> :
1496
- `--transfer-info=@preset:<psetname>` or be specified using the extended value syntax :
1497
- `--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
1498
-
1499
- If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
1500
-
1501
- If the `password` value begins with `Bearer ` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.
1502
-
1503
- #### <a id="agt_httpgw"></a>HTTP Gateway
1504
-
1505
- If it possible to send using a HTTP gateway, in case FASP is not allowed. `transfer_info` shall have a single mandatory parameter: `url`.
1506
-
1507
- Example:
1508
-
1509
- ```javascript
1510
- <%=cmd%> faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
1511
- ```
1512
-
1513
- Note that the gateway only supports transfers authorized with a token.
1514
-
1515
- #### <a id="agt_trsdk"></a>Transfer SDK
1516
-
1517
- Another possibility is to use the Transfer SDK daemon (asperatransferd).
1518
-
1519
- By default it will listen on local port `55002` on `127.0.0.1`.
1520
-
1521
- ### <a id="transferspec"></a>Transfer Specification
1522
-
1523
- Some commands lead to file transfer (upload/download), all parameters necessary for this transfer
1524
- is described in a _transfer-spec_ (Transfer Specification), such as:
1525
-
1526
- * server address
1527
- * transfer user name
1528
- * credentials
1529
- * file list
1530
- * etc...
1531
-
1532
- <%=tool%> builds a default _transfer-spec_ internally, so it is not necessary to provide additional parameters on the command line for this transfer.
1533
-
1534
- If needed, it is possible to modify or add any of the supported _transfer-spec_ parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several _transfer-spec_ parameters. Multiple `ts` options on command line are cumulative.
1535
-
1536
- It is possible to specify ascp options when the `transfer` option is set to [`direct`](#agt_direct) using the special [_transfer-spec_](#transferspec) parameter: `EX_ascp_args`. Example: `--ts=@json:'{"EX_ascp_args":["-l","100m"]}'`. This is especially useful for ascp command line parameters not supported yet in the transfer spec.
1537
-
1538
- The use of a _transfer-spec_ instead of `ascp` parameters has the advantage of:
1539
-
1540
- * common to all [Transfer Agent](#agents)
1541
- * not dependent on command line limitations (special characters...)
1542
-
1543
- A [_transfer-spec_](#transferspec) is a Hash table, so it is described on the command line with the [Extended Value Syntax](#extended).
1544
-
1545
- ### <a id="transferparams"></a>Transfer Parameters
1546
-
1547
- All standard _transfer-spec_ parameters can be specified.
1548
- [_transfer-spec_](#transferspec) can also be saved/overridden in the config file.
1549
-
1550
- References:
1551
-
1552
- * [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22)&rarr;/opt/transfers
1553
- * [Aspera Transfer SDK Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22)&rarr;Guides&rarr;API Ref&rarr;Transfer Spec V1
1554
- * [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) &rarr; search `The parameters for starting a transfer.`
1555
-
1556
- Parameters can be displayed with commands:
1557
-
1558
- ```javascript
1559
- <%=cmd%> config ascp spec
1560
- <%=cmd%> config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
1561
- ```
1562
-
1563
- Columns:
1564
-
1565
- * D=Direct (local `ascp` execution)
1566
- * N=Node API
1567
- * C=Connect Client
1568
-
1569
- `ascp` argument or environment variable is provided in description.
1570
-
1571
- Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct). (only in <%=tool%>).
1572
-
1573
- <%=spec_table%>
1574
-
1575
- #### Destination folder for transfers
1576
-
1577
- The destination folder is set by <%=tool%> by default to:
1578
-
1579
- * `.` for downloads
1580
- * `/` for uploads
1581
-
1582
- It is specified by the [_transfer-spec_](#transferspec) parameter `destination_root`.
1583
- As such, it can be modified with option: `--ts=@json:'{"destination_root":"<path>"}'`.
1584
- The option `to_folder` provides an equivalent and convenient way to change this parameter:
1585
- `--to-folder=<path>` .
1586
-
1587
- #### List of files for transfers
1588
-
1589
- When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is `sources`, the default value is `@args`, which means: take remain non used arguments (not starting with `-` as list of files.
1590
- So, by default, the list of files to transfer will be simply specified on the command line:
1591
-
1592
- ```bash
1593
- <%=cmd%> server upload ~/mysample.file secondfile
1594
- ```
1595
-
1596
- This is equivalent to:
1597
-
1598
- ```bash
1599
- <%=cmd%> server upload --sources=@args ~/mysample.file secondfile
1600
- ```
1601
-
1602
- More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
1603
-
1604
- Note that this is different from the "ascp" command line. The paradigm used by <%=tool%> is:
1605
- all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.
1606
-
1607
- For ease of use and flexibility, the list of files to transfer is specified by the option `sources`. Accepted values are:
1608
-
1609
- * `@args` : (default value) the list of files is directly provided at the end of the command line (see at the beginning of this section).
1610
-
1611
- * an [Extended Value](#extended) holding an *Array of String*. Examples:
1612
-
1613
- ```javascript
1614
- --sources=@json:'["file1","file2"]'
1615
- ```
1616
-
1617
- ```bash
1618
- --sources=@lines:@stdin:
1619
- ```
1620
-
1621
- ```ruby
1622
- --sources=@ruby:'File.read("myfilelist").split("\n")'
1623
- ```
1624
-
1625
- * `@ts` : the user provides the list of files directly in the `ts` option, in its `paths` field. Example:
1626
-
1627
- ```javascript
1628
- --sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
1629
- ```
1630
-
1631
- providing a file list directly to ascp:
1632
-
1633
- ```javascript
1634
- ... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'
1635
- ```
1636
-
1637
- * Not recommended: It is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`.
1638
-
1639
- ```javascript
1640
- --sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
1641
- ```
1642
-
1643
- This method avoids creating a copy of the file list, but has drawbacks: it applies *only* to the [`direct`](#agt_direct) transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.
1644
-
1645
- In case the file list is provided on the command line i.e. using `--sources=@args` or `--sources=<Array>` (but not `--sources=@ts`), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: `src_type`:
1646
-
1647
- * `list` : (default) the path of destination is the same as source
1648
- * `pair` : in that case, the first element is the first source, the second element is the first destination, and so on.
1649
-
1650
- Example:
1651
-
1652
- ```bash
1653
- <%=cmd%> server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1
1654
- ```
1655
-
1656
- Internally, when transfer agent [`direct`](#agt_direct) is used, a temporary file list (or pair) file is generated and provided to ascp, unless `--file-list` or `--file-pair-list` is provided in `ts` in `EX_ascp_args`.
1657
-
1658
- Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the `file id` API:
1659
-
1660
- * All files must be in the same source folder.
1661
- * If there is a single file : specify the full path
1662
- * For multiple files, specify the source folder as first item in the list followed by the list of file names.
1663
-
1664
- Source files are located on "Aspera on cloud", when :
1665
-
1666
- * the server is Aspera on Cloud, and making a download / recv
1667
- * the agent is Aspera on Cloud, and making an upload / send
1668
-
1669
- #### <a id="multisession"></a>Support of multi-session
1670
-
1671
- Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.
1672
-
1673
- * when agent=node :
1674
-
1675
- ```javascript
1676
- --ts=@json:'{"multi_session":10,"multi_session_threshold":1}'
1677
- ```
1678
-
1679
- Multi-session is directly supported by the node daemon.
1680
-
1681
- * when agent=direct :
1682
-
1683
- ```javascript
1684
- --ts=@json:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}'
1685
- ```
1686
-
1687
- Note: resume policy of "attr" may cause problems. "none" or "sparse_csum"
1688
- shall be preferred.
1689
-
1690
- Multi-session spawn is done by <%=tool%>.
1691
-
1692
- When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page).
1693
-
1694
- #### Transfer Spec Examples
1695
-
1696
- * Change target rate
1697
-
1698
- ```javascript
1699
- --ts=@json:'{"target_rate_kbps":500000}'
1700
- ```
1701
-
1702
- * Override the FASP SSH port to a specific TCP port:
1703
-
1704
- ```javascript
1705
- --ts=@json:'{"ssh_port":33002}'
1706
- ```
1707
-
1708
- * Force http fallback mode:
1709
-
1710
- ```javascript
1711
- --ts=@json:'{"http_fallback":"force"}'
1712
- ```
1713
-
1714
- * Activate progress when not activated by default on server
1715
-
1716
- ```javascript
1717
- --ts=@json:'{"precalculate_job_size":true}'
1718
- ```
1719
-
1720
- ### <a id="scheduling"></a>Lock for exclusive execution
1721
-
1722
- In some conditions, it may be desirable to ensure that <%=tool%> is not executed several times in parallel.
1723
-
1724
- For instance when <%=tool%> is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:
1725
-
1726
- * Executing instances may pile-up and kill the system
1727
- * The same file may be transferred by multiple instances at the same time.
1728
- * `preview` may generate the same files in multiple instances.
1729
-
1730
- Usually the OS native scheduler already provides some sort of protection against parallel execution:
1731
-
1732
- * The Windows scheduler does this by default
1733
- * Linux cron can leverage the utility [`flock`](https://linux.die.net/man/1/flock) to do the same:
1734
-
1735
- ```bash
1736
- /usr/bin/flock -w 0 /var/cron.lock ascli ...
1737
- ```
1738
-
1739
- <%=tool%> natively supports a locking mechanism with option `lock_port`.
1740
- (Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).
1741
-
1742
- Example:
1743
-
1744
- Run this same command in two separate terminals within less than 30 seconds:
1745
-
1746
- ```bash
1747
- <%=cmd%> config echo @ruby:'sleep(30)' --lock-port=12345
1748
- ```
1749
-
1750
- The first instance will sleep 30 seconds, the second one will immediately exit like this:
1751
-
1752
- ```bash
1753
- WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).
1754
- ```
1755
-
1756
- ### "Proven&ccedil;ale"
1757
-
1758
- `ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call *PVCL*. Several *PVCL* adapters are available, some are embedded in `ascp`
1759
- , some are provided om shared libraries and must be activated. (e.g. using `trapd`)
1760
-
1761
- The list of supported *PVCL* adapters can be retrieved with command:
1762
-
1763
- ```bash
1764
- <%=cmd%> conf ascp info
1765
- ```
1766
-
1767
- ```output
1768
- +--------------------+-----------------------------------------------------------+
1769
- | key | value |
1770
- +--------------------+-----------------------------------------------------------+
1771
- -----8<-----snip-----8<-----
1772
- | product_name | IBM Aspera SDK |
1773
- | product_version | 4.0.1.182389 |
1774
- | process | pvcl |
1775
- | shares | pvcl |
1776
- | noded | pvcl |
1777
- | faux | pvcl |
1778
- | file | pvcl |
1779
- | stdio | pvcl |
1780
- | stdio-tar | pvcl |
1781
- +--------------------+-----------------------------------------------------------+
1782
- ```
1783
-
1784
- Here we can see the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`.
1785
-
1786
- Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a pseudo "drive".
1787
-
1788
- The simplified format is:
1789
-
1790
- ```bash
1791
- <adapter>:///<sub file path>?<arg1>=<val1>&...
1792
- ```
1793
-
1794
- One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).
1795
-
1796
- ### <a id="faux_testing"></a>`faux:` for testing
1797
-
1798
- This is an extract of the man page of `ascp`. This feature is a feature of `ascp`, not <%=tool%>.
1799
-
1800
- This adapter can be used to simulate a file or a directory.
1801
-
1802
- To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:
1803
-
1804
- ```bash
1805
- faux:///filename?filesize
1806
- ```
1807
-
1808
- where:
1809
-
1810
- * `filename` is the name that will be assigned to the file on the destination
1811
- * `filesize` is the number of bytes that will be sent (in decimal).
1812
-
1813
- Note: characters `?` and `&` are shell special characters (wildcard and backround), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
1814
-
1815
- For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
1816
-
1817
- To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:
1818
-
1819
- ```bash
1820
- faux:///dirname?<arg1>=<val1>&...
1821
- ```
1822
-
1823
- where:
1824
-
1825
- * `dirname` is the folder name and can contain `/` to specify a subfolder.
1826
- * supported arguments are:
1827
-
1828
- <table>
1829
- <tr><th>name</th><th>type</th><th>default</th><th>description</th></tr>
1830
- <tr><td>count</td><td>int</td><td>mandatory</td><td>Number of files</td></tr>
1831
- <tr><td>file</td><td>string</td><td>file</td><td>Basename for files</td></tr>
1832
- <tr><td>size</td><td>int</td><td>0</td><td>Size of first file.</td></tr>
1833
- <tr><td>inc</td><td>int</td><td>0</td><td>Increment applied to determine next file size</td></tr>
1834
- <tr><td>seq</td><td>sequential<br/>random</td><td>sequential</td><td>Sequence in determining next file size</td></tr>
1835
- <tr><td>buf_init</td><td>none<br/>zero<br/>random</td><td>zero</td><td>How source data is initialized<br/>Option 'none' is not allowed for downloads.</td></tr>
1836
- </table>
1837
-
1838
- The sequence parameter is applied as follows:
1839
-
1840
- * If `seq` is `random` then each file size is:
1841
-
1842
- * size +/- (inc * rand())
1843
- * Where rand is a random number between 0 and 1
1844
- * Note that file size must not be negative, inc will be set to size if it is greater than size
1845
- * Similarly, overall file size must be less than 8*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7*2<sup>60</sup>.
1846
-
1847
- * If `seq` is `sequential` then each file size is:
1848
-
1849
- * `size + ((fileindex - 1) * inc)`
1850
- * Where first file is index 1
1851
- * So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
1852
- * As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
1853
-
1854
- Filenames generated are of the form: `<file>_<00000 ... count>_<filesize>`
1855
-
1856
- To discard data at the destination, the destination argument is set to `faux://` .
1857
-
1858
- Examples:
1859
-
1860
- * Upload 20 gibibytes of random data to file myfile to directory /Upload
1861
-
1862
- ```bash
1863
- <%=cmd%> server upload faux:///myfile\?20g --to-folder=/Upload
1864
- ```
1865
-
1866
- * Upload a file /tmp/sample but do not save results to disk (no docroot on destination)
1867
-
1868
- ```bash
1869
- <%=cmd%> server upload /tmp/sample --to-folder=faux://
1870
- ```
1871
-
1872
- * Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being `testfile` to /Upload
1873
-
1874
- ```bash
1875
- <%=cmd%> server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload
1876
- ```
1877
-
1878
- ### <a id="commands"></a>Sample Commands
1879
-
1880
- A non complete list of commands used in unit tests:
1881
-
1882
- ```bash
1883
- <%=File.read(ENV["INCL_COMMANDS"])%>
1884
- ...and more
1885
- ```
1886
-
1887
- ### <a id="usage"></a>Usage
1888
-
1889
- ```bash
1890
- <%=cmd%> -h
1891
- <%=File.read(ENV["INCL_USAGE"])%>
1892
-
1893
- ```
1894
-
1895
- Note that actions and parameter values can be written in short form.
1896
-
1897
- ## <a id="aoc"></a>Plugin: Aspera on Cloud
1898
-
1899
- Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).
1900
-
1901
- It is recommended to use the wizard to set it up, but manual configuration is also possible.
1902
-
1903
- ### <a id="aocwizard"></a>Configuration: using Wizard
1904
-
1905
- <%=tool%> provides a configuration wizard. Here is a sample invocation :
1906
-
1907
- ```bash
1908
- <%=cmd%> config wizard
1909
- option: url> https://myorg.ibmaspera.com
1910
- Detected: Aspera on Cloud
1911
- Preparing preset: aoc_myorg
1912
- Please provide path to your private RSA key, or empty to generate one:
1913
- option: pkeypath>
1914
- using existing key:
1915
- /Users/myself/.aspera/<%=cmd%>/aspera_aoc_key
1916
- Using global client_id.
1917
- option: username> john@example.com
1918
- Updating profile with new key
1919
- creating new config preset: aoc_myorg
1920
- Setting config preset as default for aspera
1921
- saving config file
1922
- Done.
1923
- You can test with:
1924
- <%=cmd%> aoc user profile show
1925
- ```
1926
-
1927
- Optionally, it is possible to create a new organization-specific "integration".
1928
- For this, specify the option: `--use-generic-client=no`.
1929
-
1930
- This will guide you through the steps to create.
1931
-
1932
- ### <a id="aocmanual"></a>Configuration: using manual setup
1933
-
1934
- If you used the wizard (recommended): skip this section.
1935
-
1936
- #### Configuration details
1937
-
1938
- Several types of OAuth authentication are supported:
1939
-
1940
- * JSON Web Token (JWT) : authentication is secured by a private key (recommended for CLI)
1941
- * Web based authentication : authentication is made by user using a browser
1942
- * URL Token : external users authentication with url tokens (public links)
1943
-
1944
- The authentication method is controlled by option `auth`.
1945
-
1946
- For a _quick start_, follow the mandatory and sufficient section: [API Client Registration](#clientreg) (auth=web) as well as [<%=prst%> for Aspera on Cloud](#aocpreset).
1947
-
1948
- For a more convenient, browser-less, experience follow the [JWT](#jwt) section (auth=jwt) in addition to Client Registration.
1949
-
1950
- In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.<%=tool%> saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.
1951
-
1952
- #### <a id="clientreg"></a>Optional: API Client Registration
1953
-
1954
- If you use the built-in client_id and client_secret, skip this and do not set them in next section.
1955
-
1956
- Else you can use a specific OAuth API client_id, the first step is to declare <%=tool%> in Aspera on Cloud using the admin interface.
1957
-
1958
- (official documentation: <https://ibmaspera.com/help/admin/organization/registering_an_api_client> ).
1959
-
1960
- Let's start by a registration with web based authentication (auth=web):
1961
-
1962
- * Open a web browser, log to your instance: e.g. `https://myorg.ibmaspera.com/`
1963
- * Go to Apps&rarr;Admin&rarr;Organization&rarr;Integrations
1964
- * Click "Create New"
1965
- * Client Name: <%=tool%>
1966
- * Redirect URIs: `http://localhost:12345`
1967
- * Origins: `localhost`
1968
- * uncheck "Prompt users to allow client to access"
1969
- * leave the JWT part for now
1970
- * Save
1971
-
1972
- Note: for web based authentication, <%=tool%> listens on a local port (e.g. specified by the redirect_uri, in this example: 12345), and the browser will provide the OAuth code there. For `<%=tool%>, HTTP is required, and 12345 is the default port.
1973
-
1974
- Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.
1975
-
1976
- #### <a id="aocpreset"></a><%=prst%> for Aspera on Cloud
1977
-
1978
- If you did not use the wizard, you can also manually create a <%=prst%> for <%=tool%> in its configuration file.
1979
-
1980
- Lets create an <%=prst%> called: `my_aoc_org` using `ask` interactive input (client info from previous step):
1981
-
1982
- ```bash
1983
- <%=cmd%> config preset ask my_aoc_org url client_id client_secret
1984
- option: url> https://myorg.ibmaspera.com/
1985
- option: client_id> BJLPObQiFw
1986
- option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
1987
- updated: my_aoc_org
1988
- ```
1989
-
1990
- (This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
1991
-
1992
- Define this <%=prst%> as default configuration for the `aspera` plugin:
1993
-
1994
- ```bash
1995
- <%=cmd%> config preset set default aoc my_aoc_org
1996
- ```
1997
-
1998
- Note: Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. Leave those default values.
1999
-
2000
- #### <a id="jwt"></a>Activation of JSON Web Token (JWT) for direct authentication
2001
-
2002
- For a Browser-less, Private Key-based authentication, use the following steps.
2003
-
2004
- ##### Key Pair Generation
2005
-
2006
- In order to use JWT for Aspera on Cloud API client authentication,
2007
- a private/public key pair must be generated (without passphrase)
2008
- This can be done using any of the following method:
2009
-
2010
- (TODO: add passphrase protection as option).
2011
-
2012
- * using the CLI:
2013
-
2014
- ```bash
2015
- <%=cmd%> config genkey ~/.aspera/<%=cmd%>/aocapikey
2016
- ```
2017
-
2018
- * `ssh-keygen`:
2019
-
2020
- ```bash
2021
- ssh-keygen -t rsa -f ~/.aspera/<%=cmd%>/aocapikey -N ''
2022
- ```
2023
-
2024
- * `openssl`
2025
-
2026
- (on some openssl implementation (mac) there is option: -nodes (no DES))
2027
-
2028
- ```bash
2029
- APIKEY=~/.aspera/<%=cmd%>/aocapikey
2030
- openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
2031
- openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
2032
- openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
2033
- rm -f ${APIKEY}.protected
2034
- ```
2035
-
2036
- ##### API Client JWT activation
2037
-
2038
- If you are not using the built-in client_id and secret, JWT needs to be authorized in Aspera on Cloud. This can be done in two manners:
2039
-
2040
- * Graphically
2041
-
2042
- * Open a web browser, log to your instance: https://myorg.ibmaspera.com/
2043
- * Go to Apps&rarr;Admin&rarr;Organization&rarr;Integrations
2044
- * Click on the previously created application
2045
- * select tab : "JSON Web Token Auth"
2046
- * Modify options if necessary, for instance: activate both options in section "Settings"
2047
- * Click "Save"
2048
-
2049
- * Using command line
2050
-
2051
- ```bash
2052
- <%=cmd%> aoc admin res client list
2053
- ```
2054
-
2055
- ```output
2056
- :............:...............:
2057
- : id : name :
2058
- :............:...............:
2059
- : BJLPObQiFw : my-client-app :
2060
- :............:...............:
2061
- ```
2062
-
2063
- ```javascript
2064
- <%=cmd%> aoc admin res client modify BJLPObQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
2065
- ```
2066
-
2067
- ```output
2068
- modified
2069
- ```
2070
-
2071
- #### User key registration
2072
-
2073
- The public key must be assigned to your user. This can be done in two manners:
2074
-
2075
- ##### Graphically
2076
-
2077
- open the previously generated public key located here: `$HOME/.aspera/<%=cmd%>/aocapikey.pub`
2078
-
2079
- * Open a web browser, log to your instance: https://myorg.ibmaspera.com/
2080
- * Click on the user's icon (top right)
2081
- * Select "Account Settings"
2082
- * Paste the _Public Key_ in the "Public Key" section
2083
- * Click on "Submit"
2084
-
2085
- ##### Using command line
2086
-
2087
- ```bash
2088
- <%=cmd%> aoc admin res user list
2089
- ```
2090
-
2091
- ```output
2092
- :........:................:
2093
- : id : name :
2094
- :........:................:
2095
- : 109952 : Tech Support :
2096
- : 109951 : LAURENT MARTIN :
2097
- :........:................:
2098
- ```
2099
-
2100
- ```ruby
2101
- <%=cmd%> aoc user profile modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/<%=cmd%>/aocapikey.pub"))}'
2102
- ```
2103
-
2104
- ```output
2105
- modified
2106
- ```
2107
-
2108
- Note: the `aspera user info show` command can be used to verify modifications.
2109
-
2110
- #### <%=prst%> modification for JWT
2111
-
2112
- To activate default use of JWT authentication for <%=tool%> using the <%=prst%>, do the following:
2113
-
2114
- * change auth method to JWT
2115
- * provide location of private key
2116
- * provide username to login as (OAuth "subject")
2117
-
2118
- Execute:
2119
-
2120
- ```bash
2121
- <%=cmd%> config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/<%=cmd%>/aocapikey --username=laurent.martin.aspera@fr.ibm.com
2122
- ```
2123
-
2124
- Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.
2125
-
2126
- After this last step, commands do not require web login anymore.
2127
-
2128
- #### <a id="aocfirst"></a>First Use
2129
-
2130
- Once client has been registered and <%=prst%> created: <%=tool%> can be used:
2131
-
2132
- ```bash
2133
- <%=cmd%> aoc files br /
2134
- ```
2135
-
2136
- ```output
2137
- Current Workspace: Default Workspace (default)
2138
- empty
2139
- ```
2140
-
2141
- ### Administration
2142
-
2143
- The `admin` command allows several administrative tasks (and require admin privilege).
2144
-
2145
- It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the `admin resource` command.
2146
-
2147
- #### Bulk creation and deletion of resource
2148
-
2149
- Bulk creation and deletion of resources are possible using option `bulk` (yes,no(default)).
2150
- In that case, the operation expects an Array of Hash instead of a simple Hash using the [Extended Value Syntax](#extended).
2151
-
2152
- #### Listing resources
2153
-
2154
- The command `aoc admin res <type> list` lists all entities of given type. It uses paging and multiple requests if necessary.
2155
-
2156
- The option `query` can be optionally used. It expects a Hash using [Extended Value Syntax](#extended), generally provided using: `--query=@json:{...}`. Values are directly sent to the API call and used as a filter on server side.
2157
-
2158
- The following parameters are supported:
2159
-
2160
- * `q` : a filter on name of resource (case insensitive, matches if value is contained in name)
2161
- * `sort`: name of fields to sort results, prefix with `-` for reverse order.
2162
- * `max` : maximum number of items to retrieve (stop pages when the maximum is passed)
2163
- * `pmax` : maximum number of pages to request (stop pages when the maximum is passed)
2164
- * `page` : native api parameter, in general do not use (added by
2165
- * `per_page` : native api parameter, number of items par api call, in general do not use
2166
- * Other specific parameters depending on resource type.
2167
-
2168
- Both `max` and `pmax` are processed internally in <%=tool%>, not included in actual API call and limit the number of successive pages requested to API. <%=tool%> will return all values using paging if not provided.
2169
-
2170
- Other parameters are directly sent as parameters to the GET request on API.
2171
-
2172
- `page` and `per_page` are normally added by <%=tool%> to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).
2173
-
2174
- `q` and `sort` are available on most resource types.
2175
-
2176
- Other parameters depend on the type of entity (refer to AoC API).
2177
-
2178
- Examples:
2179
-
2180
- * List users with `laurent` in name:
2181
-
2182
- ```javascript
2183
- <%=cmd%> aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
2184
- ```
2185
-
2186
- * List users who logged-in before a date:
2187
-
2188
- ```javascript
2189
- <%=cmd%> aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2190
- ```
2191
-
2192
- * List external users and sort in reverse alphabetical order using name:
2193
-
2194
- ```javascript
2195
- <%=cmd%> aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
2196
- ```
2197
-
2198
- Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.
2199
-
2200
- Note the option `select` can also be used to further refine selection, refer to [section earlier](#option_select).
2201
-
2202
- #### <a id="res_select"></a>Selecting a resource
2203
-
2204
- Resources are identified by a unique `id`, as well as a unique `name` (case insensitive).
2205
-
2206
- To execute an action on a specific resource, select it using one of those methods:
2207
-
2208
- * **recommended:** give id directly on command line *after the action*: `aoc admin res node show 123`
2209
- * give name on command line *after the action*: `aoc admin res node show name abc`
2210
- * provide option `id` : `aoc admin res node show --id=123`
2211
- * provide option `name` : `aoc admin res node show --name=abc`
2212
-
2213
- #### Access Key secrets
2214
-
2215
- In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required.
2216
- It is usually provided using the `secret` option.
2217
- For example in a command like:
2218
-
2219
- ```bash
2220
- <%=cmd%> aoc admin res node --id=123 --secret="secret1" v3 info
2221
- ```
2222
-
2223
- It is also possible to provide a set of secrets used on a regular basis using the [secret vault](#vault).
2224
-
2225
- #### Activity
2226
-
2227
- The activity app can be queried with:
2228
-
2229
- ```bash
2230
- <%=cmd%> aoc admin analytics transfers
2231
- ```
2232
-
2233
- It can also support filters and send notification using option `notif_to`. a template is defined using option `notif_template` :
2234
-
2235
- `mytemplate.erb`:
2236
-
2237
- ```bash
2238
- From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
2239
- To: <<%='<'%>%=ev['user_email']%>>
2240
- Subject: <%='<'%>%=ev['files_completed']%> files received
2241
-
2242
- Dear <%='<'%>%=ev[:user_email.to_s]%>,
2243
- We received <%='<'%>%=ev['files_completed']%> files for a total of <%='<'%>%=ev['transferred_bytes']%> bytes, starting with file:
2244
- <%='<'%>%=ev['content']%>
2245
-
2246
- Thank you.
2247
- ```
2248
-
2249
- The environment provided contains the following additional variable:
2250
-
2251
- * ev : all details on the transfer event
2252
-
2253
- Example:
2254
-
2255
- ```javascript
2256
- <%=cmd%> aoc admin analytics transfers --once-only=yes --lock-port=12345 \
2257
- --query=@json:'{"status":"completed","direction":"receive"}' \
2258
- --notif-to=active --notif-template=@file:mytemplate.erb
2259
- ```
2260
-
2261
- Options:
2262
-
2263
- * `once_only` keep track of last date it was called, so next call will get only new events
2264
- * `query` filter (on API call)
2265
- * `notify` send an email as specified by template, this could be places in a file with the `@file` modifier.
2266
-
2267
- Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].
2268
-
2269
- #### Transfer: Using specific transfer ports
2270
-
2271
- By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that.
2272
- The option `default_ports` ([yes]/no) allows <%=cmd%> to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
2273
-
2274
- #### Using ATS
2275
-
2276
- Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `aoc admin ats`.
2277
-
2278
- #### Example: Bulk creation of users
2279
-
2280
- ```javascript
2281
- <%=cmd%> aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
2282
- ```
2283
-
2284
- ```output
2285
- :.......:.........:
2286
- : id : status :
2287
- :.......:.........:
2288
- : 98398 : created :
2289
- : 98399 : created :
2290
- :.......:.........:
2291
- ```
2292
-
2293
- #### Example: Find with filter and delete
2294
-
2295
- ```javascript
2296
- <%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
2297
- ```
2298
-
2299
- ```output
2300
- :.......:........................:
2301
- : id : email :
2302
- :.......:........................:
2303
- : 98398 : dummyuser1@example.com :
2304
- : 98399 : dummyuser2@example.com :
2305
- :.......:........................:
2306
- ```
2307
-
2308
- ```bash
2309
- thelist=$(<%=cmd%> aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')
2310
- ```
2311
-
2312
- ```bash
2313
- echo $thelist
2314
- ```
2315
-
2316
- ```javascript
2317
- ["113501","354061"]
2318
- ```
2319
-
2320
- ```bash
2321
- <%=cmd%> aoc admin res user --bulk=yes --id=@json:"$thelist" delete
2322
- ```
2323
-
2324
- ```output
2325
- :.......:.........:
2326
- : id : status :
2327
- :.......:.........:
2328
- : 98398 : deleted :
2329
- : 98399 : deleted :
2330
- :.......:.........:
2331
- ```
2332
-
2333
- #### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
2334
-
2335
- ```ruby
2336
- <%=cmd%> aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'
2337
- ```
2338
-
2339
- To delete them use the same method as before
2340
-
2341
- #### Example: Display current user's workspaces
2342
-
2343
- ```bash
2344
- <%=cmd%> aoc user workspaces list
2345
- ```
2346
-
2347
- ```output
2348
- :......:............................:
2349
- : id : name :
2350
- :......:............................:
2351
- : 16 : Engineering :
2352
- : 17 : Marketing :
2353
- : 18 : Sales :
2354
- :......:............................:
2355
- ```
2356
-
2357
- #### Example: Create a sub access key in a "node"
2358
-
2359
- Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)
2360
-
2361
- ```bash
2362
- <%=cmd%> aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'
2363
- ```
2364
-
2365
- #### Example: Display transfer events (ops/transfer)
2366
-
2367
- ```bash
2368
- <%=cmd%> aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'
2369
- ```
2370
-
2371
- Examples of query (TODO: cleanup):
2372
-
2373
- ```javascript
2374
- {"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
2375
- ```
2376
-
2377
- ```javascript
2378
- {"tag":"aspera.files.package_id=LA8OU3p8w"}
2379
- ```
2380
-
2381
- #### Example: Display node events (events)
2382
-
2383
- ```bash
2384
- <%=cmd%> aoc admin res node --secret=_secret_ v3 events
2385
- ```
2386
-
2387
- #### Example: Display members of a workspace
2388
-
2389
- ```javascript
2390
- <%=cmd%> aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
2391
- ```
2392
-
2393
- ```output
2394
- :.............:.........:..................................:
2395
- : member_type : manager : member.email :
2396
- :.............:.........:..................................:
2397
- : user : true : john.curtis@email.com :
2398
- : user : false : laurent.martin.aspera@fr.ibm.com :
2399
- : user : false : jean.dupont@me.com :
2400
- : user : false : another.user@example.com :
2401
- : group : false : :
2402
- : user : false : aspera.user@gmail.com :
2403
- :.............:.........:..................................:
2404
- ```
2405
-
2406
- other query parameters:
2407
-
2408
- ```javascript
2409
- {"workspace_membership_through":true,"include_indirect":true}
2410
- ```
2411
-
2412
- #### Example: <a id="aoc_sample_member"></a>add all members of a workspace to another workspace
2413
-
2414
- a- Get id of first workspace
2415
-
2416
- ```bash
2417
- WS1='First Workspace'
2418
- WS1ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)
2419
- ```
2420
-
2421
- b- Get id of second workspace
2422
-
2423
- ```bash
2424
- WS2='Second Workspace'
2425
- WS2ID=$(<%=cmd%> aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)
2426
- ```
2427
-
2428
- c- Extract membership information
2429
-
2430
- ```bash
2431
- <%=cmd%> aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json
2432
- ```
2433
-
2434
- d- Convert to creation data for second workspace:
2435
-
2436
- ```bash
2437
- grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json
2438
- ```
2439
-
2440
- or, using jq:
2441
-
2442
- ```bash
2443
- jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json
2444
- ```
2445
-
2446
- e- Add members to second workspace
2447
-
2448
- ```bash
2449
- <%=cmd%> aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json
2450
- ```
2451
-
2452
- #### Example: Get users who did not log since a date
2453
-
2454
- ```javascript
2455
- <%=cmd%> aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
2456
- ```
2457
-
2458
- ```output
2459
- :...............................:
2460
- : email :
2461
- :...............................:
2462
- : John.curtis@acme.com :
2463
- : Jean.Dupont@tropfort.com :
2464
- :...............................:
2465
- ```
2466
-
2467
- #### Example: List "Limited" users
2468
-
2469
- ```javascript
2470
- <%=cmd%> aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
2471
- ```
2472
-
2473
- #### Example: Perform a multi Gbps transfer between two remote shared folders
2474
-
2475
- In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.
2476
-
2477
- First, setup the environment (skip if already done)
2478
-
2479
- ```bash
2480
- <%=cmd%> conf wizard --url=https://sedemo.ibmaspera.com --username=laurent.martin.aspera@fr.ibm.com
2481
- ```
2482
-
2483
- ```output
2484
- Detected: Aspera on Cloud
2485
- Preparing preset: aoc_sedemo
2486
- Using existing key:
2487
- /Users/laurent/.aspera/<%=cmd%>/aspera_aoc_key
2488
- Using global client_id.
2489
- Please Login to your Aspera on Cloud instance.
2490
- Navigate to your "Account Settings"
2491
- Check or update the value of "Public Key" to be:
2492
- -----BEGIN PUBLIC KEY-----
2493
- SOME PUBLIC KEY PEM DATA HERE
2494
- -----END PUBLIC KEY-----
2495
- Once updated or validated, press enter.
2496
-
2497
- creating new config preset: aoc_sedemo
2498
- Setting config preset as default for aspera
2499
- saving config file
2500
- Done.
2501
- You can test with:
2502
- <%=cmd%> aoc user profile show
2503
- ```
2504
-
2505
- This creates the option preset "aoc_&lt;org name&gt;" to allow seamless command line access and sets it as default for aspera on cloud.
2506
-
2507
- Then, create two shared folders located in two regions, in your files home, in a workspace.
2508
-
2509
- Then, transfer between those:
2510
-
2511
- ```javascript
2512
- <%=cmd%> -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
2513
- ```
2514
-
2515
- #### Example: create registration key to register a node
2516
-
2517
- ```javascript
2518
- <%=cmd%> aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
2519
- ```
2520
-
2521
- ```output
2522
- jfqslfdjlfdjfhdjklqfhdkl
2523
- ```
2524
-
2525
- #### Example: delete all registration keys
2526
-
2527
- ```bash
2528
- <%=cmd%> aoc admin res client list --fields=id --format=csv|<%=cmd%> aoc admin res client delete --bulk=yes --id=@lines:@stdin:
2529
- ```
2530
-
2531
- ```output
2532
- +-----+---------+
2533
- | id | status |
2534
- +-----+---------+
2535
- | 99 | deleted |
2536
- | 100 | deleted |
2537
- | 101 | deleted |
2538
- | 102 | deleted |
2539
- +-----+---------+
2540
- ```
2541
-
2542
- #### Example: Create a node
2543
-
2544
- AoC nodes as actually composed with two related entities:
2545
-
2546
- * An access key created on the Transfer Server (HSTS/ATS)
2547
- * a `node` resource in the AoC application.
2548
-
2549
- The web UI allows creation of both entities in one shot but not the CLI for more flexibility.
2550
- Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.
2551
-
2552
- So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):
2553
-
2554
- * create the access key on ATS
2555
-
2556
- ```javascript
2557
- <%=cmd%> aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
2558
- ```
2559
-
2560
- Take a note of the randomly generated `id` and `secret`.
2561
-
2562
- * Retrieve the ATS node address
2563
-
2564
- ```bash
2565
- <%=cmd%> aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv --transpose-single=no
2566
- ```
2567
-
2568
- * Create the node entity
2569
-
2570
- ```javascript
2571
- <%=cmd%> aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
2572
- ```
2573
-
2574
- Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
2575
-
2576
- ### Packages
2577
-
2578
- The webmail-like application.
2579
-
2580
- #### Send a Package
2581
-
2582
- General syntax:
2583
-
2584
- ```bash
2585
- <%=cmd%> aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]
2586
- ```
2587
-
2588
- Notes:
2589
-
2590
- * The `value` option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
2591
- * List allowed shared inbox destinations with: `<%=cmd%> aoc packages shared_inboxes list`
2592
- * Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: user or shared inbox.
2593
- * Provide either ids as expected by API: `"recipients":[{"type":"dropbox","id":"1234"}]`
2594
- * or just names: `"recipients":[{"The Dest"}]` . <%=cmd%> will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
2595
- * If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
2596
- * if the option `new_user_option` is `@json:{"package_contact":true}` (default), then a public link is sent and the external user does not need to create an account
2597
- * if the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace
2598
-
2599
- #### Example: Send a package with one file to two users, using their email
2600
-
2601
- ```javascript
2602
- <%=cmd%> aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
2603
- ```
2604
-
2605
- #### Example: Send a package to a shared inbox with metadata
2606
-
2607
- ```javascript
2608
- <%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox With Meta"],"metadata":{"Project Id":"123","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' ~/Documents/Samples/200KB.1
2609
- ```
2610
-
2611
- It is also possible to use identifiers and API parameters:
2612
-
2613
- ```javascript
2614
- <%=cmd%> aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":[{"type":"dropbox","id":"12345"}],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
2615
- ```
2616
-
2617
- #### Example: List packages in a given shared inbox
2618
-
2619
- When user packages are listed, the following query is used:
2620
-
2621
- ```javascript
2622
- {"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}
2623
- ```
2624
-
2625
- To list packages in a shared inbox, the query has to be specified with withe the shared inbox by name or its identifier. Additionnal parameters can be specified, as supported by the API (to find out available filters, consult the API definition, or use the web interface in developer mode). The current workspace is added unless specified in the query.
2626
-
2627
- Using shared inbox name:
2628
-
2629
- ```javascript
2630
- <%=cmd%> aoc packages list --query=@json:'{"dropbox_name":"My Shared Inbox","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
2631
- ```
2632
-
2633
- Using shared inbox identifier: first retrieve the id of the shared inbox, and then list packages with the appropriate filter.
2634
-
2635
- ```bash
2636
- shbxid=$(<%=cmd%> aoc packages shared_inboxes show name 'My Shared Inbox' --format=csv --display=data --fields=id --transpose-single=no)
2637
- ```
2638
-
2639
- ```javascript
2640
- <%=cmd%> aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
2641
- ```
2642
-
2643
- #### <a id="aoccargo"></a>Receive new packages only (Cargo)
2644
-
2645
- It is possible to automatically download new packages, like using Aspera Cargo:
2646
-
2647
- ```bash
2648
- <%=cmd%> aoc packages recv --id=ALL --once-only=yes --lock-port=12345
2649
- ```
2650
-
2651
- * `--id=ALL` (case sensitive) will download all packages
2652
- * `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder
2653
- * `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel
2654
-
2655
- Typically, one would execute this command on a regular basis, using the method of your choice:
2656
-
2657
- * Windows: [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page)
2658
- * Linux/Unix: [cron](https://www.man7.org/linux/man-pages/man5/crontab.5.html)
2659
- * etc...
2660
-
2661
- ### Files
2662
-
2663
- Folder sharing app.
2664
-
2665
- #### Download Files
2666
-
2667
- Download of files is straightforward with a specific syntax for the `aoc files download` action: Like other commands the source file list is provided as a list with the `sources` option. Nevertheless, consider this:
2668
-
2669
- * if only one source is provided, it is downloaded
2670
- * if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).
2671
-
2672
- #### Shared folders
2673
-
2674
- * list shared folders in node
2675
-
2676
- ```bash
2677
- <%=cmd%> aoc admin res node --id=8669 shared_folders
2678
- ```
2679
-
2680
- * list shared folders in workspace
2681
-
2682
- ```bash
2683
- <%=cmd%> aoc admin res workspace --id=10818 shared_folders
2684
- ```
2685
-
2686
- * list members of shared folder
2687
-
2688
- ```bash
2689
- <%=cmd%> aoc admin res node --id=8669 v4 perm 82 show
2690
- ```
2691
-
2692
- #### Cross Organization transfers
2693
-
2694
- It is possible to transfer files directly between organizations without having to first download locally and then upload...
2695
-
2696
- Although optional, the creation of <%=prst%> is recommended to avoid placing all parameters in the command line.
2697
-
2698
- Procedure to send a file from org1 to org2:
2699
-
2700
- * Get access to Organization 1 and create a <%=prst%>: e.g. `org1`, for instance, use the [Wizard](#aocwizard)
2701
- * Check that access works and locate the source file e.g. `mysourcefile`, e.g. using command `files browse`
2702
- * Get access to Organization 2 and create a <%=prst%>: e.g. `org2`
2703
- * Check that access works and locate the destination folder `mydestfolder`
2704
- * execute the following:
2705
-
2706
- ```bash
2707
- <%=cmd%> -Porg1 aoc files node_info /mydestfolder --format=json --display=data | <%=cmd%> -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:
2708
- ```
2709
-
2710
- Explanation:
2711
-
2712
- * `-Porg1 aoc` use Aspera on Cloud plugin and load credentials for `org1`
2713
- * `files node_info /mydestfolder` generate transfer information including node api credential and root id, suitable for the next command
2714
- * `--format=json` format the output in JSON (instead of default text table)
2715
- * `--display=data` display only the result, and remove other information, such as workspace name
2716
- * `|` the standard output of the first command is fed into the second one
2717
- * `-Porg2 aoc` use Aspera on Cloud plugin and load credentials for `org2`
2718
- * `files upload mysourcefile` upload the file named `mysourcefile` (located in `org1`)
2719
- * `--transfer=node` use transfer agent type `node` instead of default [`direct`](#agt_direct)
2720
- * `--transfer-info=@json:@stdin:` provide `node` transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input
2721
-
2722
- #### Find Files
2723
-
2724
- The command `aoc files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
2725
-
2726
- The expression can be of 3 formats:
2727
-
2728
- * empty (default) : all files, equivalent to value: `exec:true`
2729
- * not starting with `exec:` : the expression is a regular expression, using [Ruby Regex](https://ruby-doc.org/core/Regexp.html) syntax. equivalent to value: `exec:f['name'].match(/expression/)`
2730
-
2731
- For instance, to find files with a special extension, use `--value='\.myext$'`
2732
-
2733
- * starting with `exec:` : the Ruby code after the prefix is executed for each entry found. The entry variable name is `f`. The file is displayed if the result of the expression is true;
2734
-
2735
- Examples of expressions: (using like this: `--value=exec:'<expression>'`)
2736
-
2737
- * Find files more recent than 100 days
2738
-
2739
- ```bash
2740
- f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
2741
- ```
2742
-
2743
- * Find files older than 1 year on a given node and store in file list
2744
-
2745
- ```bash
2746
- <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
2747
- ```
2748
-
2749
- * Delete the files, one by one
2750
-
2751
- ```bash
2752
- cat my_file_list.txt|while read path;do echo <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
2753
- ```
2754
-
2755
- * Delete the files in bulk
2756
-
2757
- ```bash
2758
- cat my_file_list.txt | <%=cmd%> aoc admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
2759
- ```
2760
-
2761
- ## <a id="ats"></a>Plugin: Aspera Transfer Service
2762
-
2763
- ATS is usable either :
2764
-
2765
- * from an AoC subscription : <%=cmd%> aoc admin ats : use AoC authentication
2766
-
2767
- * or from an IBM Cloud subscription : <%=cmd%> ats : use IBM Cloud API key authentication
2768
-
2769
- ### IBM Cloud ATS : creation of api key
2770
-
2771
- This section is about using ATS with an IBM cloud subscription.
2772
- If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.
2773
-
2774
- First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
2775
-
2776
- ```bash
2777
- ibmcloud iam api-key-create mykeyname -d 'my sample key'
2778
- ```
2779
-
2780
- ```output
2781
- OK
2782
- API key mykeyname was created
2783
-
2784
- Please preserve the API key! It cannot be retrieved after it's created.
2785
-
2786
- Name mykeyname
2787
- Description my sample key
2788
- Created At 2019-09-30T12:17+0000
2789
- API Key my_secret_api_key_here_8f8d9fdakjhfsashjk678
2790
- Locked false
2791
- UUID ApiKey-05b8fadf-e7fe-4bc4-93a9-6fd348c5ab1f
2792
- ```
2793
-
2794
- References:
2795
-
2796
- * [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
2797
- * [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
2798
-
2799
- Then, to register the key by default for the ats plugin, create a preset. Execute:
2800
-
2801
- ```bash
2802
- <%=cmd%> config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678
2803
- ```
2804
-
2805
- ```bash
2806
- <%=cmd%> config preset set default ats my_ibm_ats
2807
- ```
2808
-
2809
- ```bash
2810
- <%=cmd%> ats api_key instances
2811
- ```
2812
-
2813
- ```output
2814
- +--------------------------------------+
2815
- | instance |
2816
- +--------------------------------------+
2817
- | aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
2818
- +--------------------------------------+
2819
- ```
2820
-
2821
- ```bash
2822
- <%=cmd%> config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
2823
- ```
2824
-
2825
- ```bash
2826
- <%=cmd%> ats api_key create
2827
- ```
2828
-
2829
- ```output
2830
- +--------+----------------------------------------------+
2831
- | key | value |
2832
- +--------+----------------------------------------------+
2833
- | id | ats_XXXXXXXXXXXXXXXXXXXXXXXX |
2834
- | secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
2835
- +--------+----------------------------------------------+
2836
- <%=cmd%> config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2837
- ```
2838
-
2839
- ### Misc. Examples
2840
-
2841
- Example: create access key on IBM Cloud (softlayer):
2842
-
2843
- ```javascript
2844
- <%=cmd%> ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
2845
- ```
2846
-
2847
- Example: create access key on AWS:
2848
-
2849
- ```javascript
2850
- <%=cmd%> ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'
2851
- ```
2852
-
2853
- Example: create access key on Azure SAS:
2854
-
2855
- ```javascript
2856
- <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
2857
- ```
2858
-
2859
- (Note that the blob name is mandatory after server address and before parameters. and that parameter sr=c is mandatory.)
2860
-
2861
- Example: create access key on Azure:
2862
-
2863
- ```javascript
2864
- <%=cmd%> ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
2865
- ```
2866
-
2867
- delete all my access keys:
2868
-
2869
- ```bash
2870
- for k in $(<%=cmd%> ats access_key list --field=id --format=csv);do <%=cmd%> ats access_key id $k delete;done
2871
- ```
2872
-
2873
- The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint.
2874
-
2875
- ## Plugin: IBM Aspera High Speed Transfer Server (transfer)
2876
-
2877
- This plugin uses SSH as a session protocol (using commands `ascp` and `ascmd`) and does not use the node API.
2878
- It is the legacy way of accessing an Aspera Server, often used for server to server transfers.
2879
- Modern mode is to use the node API and transfer tokens.
2880
-
2881
- ### Authentication
2882
-
2883
- Both password and SSH keys auth are supported.
2884
-
2885
- If username is not provided, the default transfer user `xfer` is used.
2886
-
2887
- If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:
2888
-
2889
- ```javascript
2890
- <%=cmd%> server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'
2891
- ```
2892
-
2893
- Multiple SSH key paths can be provided.
2894
- The value of the parameter `ssh_keys` can be a single value or an array.
2895
- Each value is a path to a private key and is expanded (`~` is replaced with the user's home folder).
2896
-
2897
- Examples:
2898
-
2899
- ```bash
2900
- <%=cmd%> server --ssh-keys=~/.ssh/id_rsa
2901
- <%=cmd%> server --ssh-keys=@list:,~/.ssh/id_rsa
2902
- <%=cmd%> server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
2903
- ```
2904
-
2905
- The underlying ssh library `net::ssh` provides several options that may be used depending on environment.
2906
- By default the ssh library expect that an ssh-agent is running.
2907
-
2908
- On Linux, if you get an error message such as:
2909
-
2910
- ```bash
2911
- ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured
2912
- ```
2913
-
2914
- or on Windows:
2915
-
2916
- ```bash
2917
- ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
2918
- ```
2919
-
2920
- This means that you don't have such an ssh agent running, then:
2921
-
2922
- * check env var: `SSH_AGENT_SOCK`
2923
- * check if the ssh key is protected with a passphrase
2924
- * [check the manual](https://net-ssh.github.io/ssh/v1/chapter-2.html#s2)
2925
- * To disable use of `ssh-agent`, use the option `ssh_option` like this:
2926
-
2927
- ```bash
2928
- <%=cmd%> server --ssh-options=@ruby:'{use_agent: false}' ...
2929
- ```
2930
-
2931
- This can also be set as default using a global preset.
2932
-
2933
- ### Example
2934
-
2935
- One can test the `server` application using the well known demo server:
2936
-
2937
- ```bash
2938
- <%=cmd%> config initdemo
2939
- <%=cmd%> server browse /aspera-test-dir-large
2940
- <%=cmd%> server download /aspera-test-dir-large/200MB
2941
- ```
2942
-
2943
- `initdemo` creates a <%=prst%> `demoserver` and set it as default for plugin `server`.
2944
-
2945
- ## Plugin: IBM Aspera High Speed Transfer Server (node)
2946
-
2947
- This plugin gives access to capabilities provided by HSTS node API.
2948
-
2949
- ### File Operations
2950
-
2951
- It is possible to:
2952
-
2953
- * browse
2954
- * transfer (upload / download)
2955
- * ...
2956
-
2957
- For transfers, it is possible to control how transfer is authorized using option: `token_type`:
2958
-
2959
- * `aspera` : api `<upload|download>_setup` is called to create the transfer spec including the Aspera token
2960
- * `basic` : transfer spec is created like this:
2961
-
2962
- ```javascript
2963
- {
2964
- "remote_host": address of node url,
2965
- "remote_user": "xfer",
2966
- "ssh_port": 33001,
2967
- "token": "Basic <base 64 encoded user/pass>",
2968
- "direction": send/recv
2969
- }
2970
- ```
2971
-
2972
- * `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
2973
-
2974
- ### Central
2975
-
2976
- The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.
2977
-
2978
- Filtering can be applied:
2979
-
2980
- ```bash
2981
- <%=cmd%> node central file list
2982
- ```
2983
-
2984
- by providing the `validator` option, offline transfer validation can be done.
2985
-
2986
- ### FASP Stream
2987
-
2988
- It is possible to start a FASPStream session using the node API:
2989
-
2990
- Use the "node stream create" command, then arguments are provided as a [_transfer-spec_](#transferspec).
2991
-
2992
- ```javascript
2993
- <%=cmd%> node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream
2994
- ```
2995
-
2996
- ### Watchfolder
2997
-
2998
- Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/entsrv/3.7.4/es_admin_linux/webhelp/index.html#watchfolder_external/dita/json_conf.html) for watch folder creation.
2999
-
3000
- <%=tool%> supports remote operations through the node API. Operations are:
3001
-
3002
- * Start watchd and watchfolderd services running as a system user having access to files
3003
- * configure a watchfolder to define automated transfers
3004
-
3005
- ```javascript
3006
- <%=cmd%> node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
3007
- <%=cmd%> node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
3008
- <%=cmd%> node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
3009
- ```
3010
-
3011
- ### Out of Transfer File Validation
3012
-
3013
- Follow the Aspera Transfer Server configuration to activate this feature.
3014
-
3015
- ```javascript
3016
- <%=cmd%> node central file list --validator=<%=cmd%> --data=@json:'{"file_transfer_filter":{"max_result":1}}'
3017
- ```
3018
-
3019
- ```output
3020
- :..............:..............:............:......................................:
3021
- : session_uuid : file_id : status : path :
3022
- :..............:..............:............:......................................:
3023
- : 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
3024
- :..............:..............:............:......................................:
3025
- ```
3026
-
3027
- ```javascript
3028
- <%=cmd%> node central file update --validator=<%=cmd%> --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
3029
- ```
3030
-
3031
- ```output
3032
- updated
3033
- ```
3034
-
3035
- ### Example: SHOD to ATS
3036
-
3037
- Scenario: Access to a "Shares on Demand" (SHOD) server on AWS is provided by a partner.
3038
- We need to transfer files from this third party SHOD instance into our Azure BLOB storage.
3039
- Simply create an "Aspera Transfer Service" instance (https://ts.asperasoft.com), which provides access to the node API.
3040
- Then create a configuration for the "SHOD" instance in the configuration file: in section "shares", a configuration named: awsshod.
3041
- Create another configuration for the Azure ATS instance: in section "node", named azureats.
3042
- Then execute the following command:
3043
-
3044
- ```bash
3045
- <%=cmd%> node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats
3046
- ```
3047
-
3048
- This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.
3049
-
3050
- ### Create access key
3051
-
3052
- ```javascript
3053
- <%=cmd%> node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'
3054
- ```
3055
-
3056
- ## Plugin: IBM Aspera Faspex5
3057
-
3058
- 3 authentication methods are supported:
3059
-
3060
- * jwt
3061
- * web
3062
- * boot
3063
-
3064
- For JWT, create an API client in Faspex with jwt support, and use: `--auth=jwt`.
3065
-
3066
- For web method, create an API client in Faspex, and use: --auth=web
3067
-
3068
- For boot method: (will be removed in future)
3069
-
3070
- * open a browser
3071
- * start developer mode
3072
- * login to faspex 5
3073
- * find the first API call with `Authorization` token, and copy it (kind of base64 long string)
3074
-
3075
- Use it as password and use `--auth=boot`.
3076
-
3077
- ```bash
3078
- <%=cmd%> conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...
3079
- ```
3080
-
3081
- Ready to use Faspex5 with CLI.
3082
-
3083
- ## Plugin: IBM Aspera Faspex (4.x)
3084
-
3085
- Notes:
3086
-
3087
- * The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
3088
- * For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
3089
-
3090
- ### Listing Packages
3091
-
3092
- Command: `faspex package list`
3093
-
3094
- #### Option `box`
3095
-
3096
- By default it looks in box `inbox`, but the following boxes are also supported: `archive` and `sent`, selected with option `box`.
3097
-
3098
- #### Option `recipient`
3099
-
3100
- A user can receive a package because the recipient is:
3101
-
3102
- * the user himself (default)
3103
- * the user is part of a dropbox or a workgroup (select with option `recipient` with value `*<name of WG or DB>`
3104
-
3105
- #### Option `query`
3106
-
3107
- As inboxes may be large, it is possible to use the following query parameters:
3108
-
3109
- * `count` : (native) number items in one API call (default=0, equivalent to 10)
3110
- * `page` : (native) id of page in call (default=0)
3111
- * `startIndex` : (native) index of item to start, default=0, oldest index=0
3112
- * `max` : maximum number of items
3113
- * `pmax` : maximum number of pages
3114
-
3115
- (SQL query is `LIMIT <startIndex>, <count>`)
3116
-
3117
- The API is listed in [Faspex 4 API Reference](https://developer.ibm.com/apis/catalog/?search=faspex) under "Services (API v.3)".
3118
-
3119
- If no parameter `max` or `pmax` is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: `count` and `page`). By default page is `0` (`10`), it can be increased to have less calls.
3120
-
3121
- #### Example: list packages in dropbox
3122
-
3123
- ```javascript
3124
- <%=cmd%> faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
3125
- ```
3126
-
3127
- List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox `*my_dropbox`.
3128
-
3129
- ### Receiving a Package
3130
-
3131
- The command is `package recv`, possible methods are:
3132
-
3133
- * provide a package id with option `id`
3134
- * provide a public link with option `link`
3135
- * provide a `faspe:` URI with option `link`
3136
-
3137
- ```bash
3138
- <%=cmd%> faspex package recv --id=12345
3139
- <%=cmd%> faspex package recv --link=faspe://...
3140
- ```
3141
-
3142
- If the package is in a specific dropbox, add option `recipient` for both the `list` and `recv` commands.
3143
-
3144
- ```bash
3145
- <%=cmd%> faspex package list --recipient='*thedropboxname'
3146
- ```
3147
-
3148
- if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
3149
-
3150
- ### Sending a Package
3151
-
3152
- The command is `faspex package send`. Package information (title, note, metadata, options) is provided in option `delivery_info`. (Refer to Faspex API).
3153
-
3154
- Example:
3155
-
3156
- ```javascript
3157
- <%=cmd%> faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
3158
- ```
3159
-
3160
- If the recipient is a dropbox, just provide the name of the dropbox in `recipients`: `"recipients":["My Dropbox Name"]`
3161
-
3162
- Additional optional parameters in `delivery_info`:
3163
-
3164
- * Package Note: : `"note":"note this and that"`
3165
- * Package Metadata: `"metadata":{"Meta1":"Val1","Meta2":"Val2"}`
3166
-
3167
- ### Email notification on transfer
3168
-
3169
- Like for any transfer, a notification can be sent by email using parameters: `notif_to` and `notif_template` .
3170
-
3171
- Example:
3172
-
3173
- ```javascript
3174
- <%=cmd%> faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>\nTo: <<%='<'%>%=to%>>\nSubject: Package sent: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%='<'%>%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
3175
- ```
3176
-
3177
- In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: `@file:`
3178
-
3179
- ### Operation on dropboxes
3180
-
3181
- Example:
3182
-
3183
- ```javascript
3184
- <%=cmd%> faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
3185
- <%=cmd%> faspex v4 dropbox list
3186
- <%=cmd%> faspex v4 dropbox delete --id=36
3187
- ```
3188
-
3189
- ### Remote sources
3190
-
3191
- Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this,
3192
- the node API is used, for this it is required to add a section ":storage" that links
3193
- a storage name to a node config and sub path.
3194
-
3195
- Example:
3196
-
3197
- ```yaml
3198
- my_faspex_conf:
3199
- url: https://10.25.0.3/aspera/faspex
3200
- username: admin
3201
- password: MyPassword
3202
- storage:
3203
- testlaurent:
3204
- node: "@preset:my_faspex_node"
3205
- path: /myfiles
3206
- my_faspex_node:
3207
- url: https://10.25.0.3:9092
3208
- username: node_faspex
3209
- password: MyPassword
3210
- ```
3211
-
3212
- In this example, a faspex storage named "testlaurent" exists in Faspex, and is located
3213
- under the docroot in "/myfiles" (this must be the same as configured in Faspex).
3214
- The node configuration name is "my_faspex_node" here.
3215
-
3216
- Note: the v4 API provide an API for nodes and shares.
3217
-
3218
- ### Automated package download (cargo)
3219
-
3220
- It is possible to tell <%=tool%> to download newly received packages, much like the official
3221
- cargo client, or drive. Refer to the [same section](#aoccargo) in the Aspera on Cloud plugin:
3222
-
3223
- ```bash
3224
- <%=cmd%> faspex packages recv --id=ALL --once-only=yes --lock-port=12345
3225
- ```
3226
-
3227
- ## Plugin: IBM Aspera Shares
3228
-
3229
- Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and 2)
3230
-
3231
- In Shares2, users, groups listing are paged, to display sequential pages:
3232
-
3233
- ```bash
3234
- for p in 1 2 3;do <%=cmd%> shares2 admin users list --value=@json:'{"page":'$p'}';done
3235
- ```
3236
-
3237
- ## Plugin: IBM Cloud Object Storage
3238
-
3239
- The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
3240
- It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS).
3241
- Available ATS regions: [https://status.aspera.io](https://status.aspera.io)
3242
-
3243
- There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.
3244
-
3245
- ### Using endpoint, apikey and Resource Instance ID (CRN)
3246
-
3247
- If you have those parameters already, then following options shall be provided:
3248
-
3249
- * `bucket` bucket name
3250
- * `endpoint` storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
3251
- * `apikey` API Key
3252
- * `crn` resource instance id
3253
-
3254
- For example, let us create a default configuration:
3255
-
3256
- ```bash
3257
- <%=cmd%> conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
3258
- <%=cmd%> conf id default set cos mycos
3259
- ```
3260
-
3261
- Then, jump to the transfer example.
3262
-
3263
- ### Using service credential file
3264
-
3265
- If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:
3266
-
3267
- Navigation Menu &rarr; Resource List &rarr; Storage &rarr; Cloud Object Storage &rarr; Service Credentials &rarr; &lt;select or create credentials&gt; &rarr; view credentials &rarr; copy
3268
-
3269
- Then save the copied value to a file, e.g. : `$HOME/cos_service_creds.json`
3270
-
3271
- or using the IBM Cloud CLI:
3272
-
3273
- ```bash
3274
- ibmcloud resource service-keys
3275
- ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
3276
- ```
3277
-
3278
- (if you don't have `jq` installed, extract the structure as follows)
3279
-
3280
- It consists in the following structure:
3281
-
3282
- ```javascript
3283
- {
3284
- "apikey": "xxxxxxx.....",
3285
- "cos_hmac_keys": {
3286
- "access_key_id": "xxxxxxx.....",
3287
- "secret_access_key": "xxxxxxx....."
3288
- },
3289
- "endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints",
3290
- "iam_apikey_description": "my description ...",
3291
- "iam_apikey_name": "my key name",
3292
- "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
3293
- "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxxx.....",
3294
- "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/xxxxxxx....."
3295
- }
3296
- ```
3297
-
3298
- The field `resource_instance_id` is for option `crn`
3299
-
3300
- The field `apikey` is for option `apikey`
3301
-
3302
- (If needed: endpoints for regions can be found by querying the `endpoints` URL.)
3303
-
3304
- The required options for this method are:
3305
-
3306
- * `bucket` bucket name
3307
- * `region` bucket region, e.g. eu-de
3308
- * `service_credentials` see below
3309
-
3310
- For example, let us create a default configuration:
3311
-
3312
- ```bash
3313
- <%=cmd%> conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
3314
- <%=cmd%> conf id default set cos mycos
3315
- ```
3316
-
3317
- ### Operations, transfers
3318
-
3319
- Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).
3320
-
3321
- A subset of `node` plugin operations are supported, basically node API:
3322
-
3323
- ```bash
3324
- <%=cmd%> cos node info
3325
- <%=cmd%> cos node upload 'faux:///sample1G?1g'
3326
- ```
3327
-
3328
- Note: we generate a dummy file `sample1G` of size 2GB using the `faux` PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.
3329
-
3330
- ## Plugin: IBM Aspera Sync
3331
-
3332
- A basic plugin to start an "async" using <%=tool%>.
3333
- The main advantage is the possibility to start from ma configuration file, using <%=tool%> standard options.
3334
-
3335
- ## Plugin: Preview
3336
-
3337
- The `preview` generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application.
3338
- This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root".
3339
- Several parameters can be used to tune several aspects:
3340
-
3341
- * methods for detection of new files needing generation
3342
- * methods for generation of video preview
3343
- * parameters for video handling
3344
-
3345
- ### Aspera Server configuration
3346
-
3347
- Specify the previews folder as shown in:
3348
-
3349
- <https://ibmaspera.com/help/admin/organization/installing_the_preview_maker>
3350
-
3351
- By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute:
3352
-
3353
- ```bash
3354
- PATH=/opt/aspera/bin:$PATH
3355
-
3356
- asconfigurator -x "server;preview_dir,previews"
3357
- asnodeadmin --reload
3358
- ```
3359
-
3360
- Note: the configuration `preview_dir` is *relative* to the storage root, no need leading or trailing `/`. In general just set the value to `previews`
3361
-
3362
- If another folder is configured on the HSTS, then specify it to <%=tool%> using the option `previews_folder`.
3363
-
3364
- The HSTS node API limits any preview file to a parameter: `max_request_file_create_size_kb` (1 KB is 1024 bytes).
3365
- This size is internally capped to `1<<24` Bytes (16777216) , i.e. 16384 KBytes.
3366
-
3367
- To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`:
3368
-
3369
- ```bash
3370
- asuserdata -a | grep max_request_file_create_size_kb
3371
-
3372
- max_request_file_create_size_kb: "1024"
3373
-
3374
- asconfigurator -x "server; max_request_file_create_size_kb,16384"
3375
- ```
3376
-
3377
- If you use a value different than 16777216, then specify it using option `max_size`.
3378
-
3379
- Note: the HSTS parameter (max_request_file_create_size_kb) is in *kiloBytes* while the generator parameter is in *Bytes* (factor of 1024).
3380
-
3381
- ### <a id="prev_ext"></a>External tools: Linux
3382
-
3383
- The tool requires the following external tools available in the `PATH`:
3384
-
3385
- * ImageMagick : `convert` `composite`
3386
- * OptiPNG : `optipng`
3387
- * FFmpeg : `ffmpeg` `ffprobe`
3388
- * Libreoffice : `libreoffice`
3389
- * ruby gem `mimemagic`
3390
-
3391
- Here shown on Redhat/CentOS.
3392
-
3393
- Other OSes should work as well, but are note tested.
3394
-
3395
- To check if all tools are found properly, execute:
3396
-
3397
- ```bash
3398
- <%=cmd%> preview check
3399
- ```
3400
-
3401
- #### mimemagic
3402
-
3403
- To benefit from extra mime type detection install gem mimemagic:
3404
-
3405
- ```bash
3406
- gem install mimemagic
3407
- ```
3408
-
3409
- or to install an earlier version if any problem:
3410
-
3411
- ```bash
3412
- gem install mimemagic -v '~> 0.3.0'
3413
- ```
3414
-
3415
- To use it, set option `mimemagic` to `yes`: `--mimemagic=yes`
3416
-
3417
- If not used, Mime type used for conversion is the one provided by the node API.
3418
-
3419
- If used, it the `preview` command will first analyze the file content using mimemagic, and if no match, will try by extension.
3420
-
3421
- #### Image: ImageMagick and optipng
3422
-
3423
- ```bash
3424
- yum install -y ImageMagick optipng
3425
- ```
3426
-
3427
- #### Video: FFmpeg
3428
-
3429
- The easiest method is to download and install the latest released version of ffmpeg with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/)
3430
-
3431
- ```bash
3432
- curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)
3433
- ```
3434
-
3435
- #### Office: Unoconv and Libreoffice
3436
-
3437
- If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: `--skip-types=office`
3438
-
3439
- The generation of preview in based on the use of `unoconv` and `libreoffice`
3440
-
3441
- * CentOS 8
3442
-
3443
- ```bash
3444
- dnf install unoconv
3445
- ```
3446
-
3447
- * Amazon Linux
3448
-
3449
- ```bash
3450
- amazon-linux-extras enable libreoffice
3451
- yum clean metadata
3452
- yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
3453
- wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
3454
- mv unoconv /usr/bin
3455
- chmod a+x /usr/bin/unoconv
3456
- ```
3457
-
3458
- ### Configuration
3459
-
3460
- The preview generator is run as a user, preferably a regular user (not root). When using object storage, any user can be used, but when using local storage it is usually better to use the user `xfer`, as uploaded files are under this identity: this ensures proper access rights. (we will assume this)
3461
-
3462
- Like any <%=tool%> commands, parameters can be passed on command line or using a configuration <%=prst%>. The configuration file must be created with the same user used to run so that it is properly used on runtime.
3463
-
3464
- Note that the `xfer` user has a special protected shell: `aspshell`, so changing identity requires specification of alternate shell:
3465
-
3466
- ```bash
3467
- su -s /bin/bash - xfer
3468
-
3469
- <%=cmd%> config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
3470
-
3471
- <%=cmd%> config preset set default preview previewconf
3472
- ```
3473
-
3474
- Here we assume that Office file generation is disabled, else remove this option.
3475
- `lock_port` prevents concurrent execution of generation when using a scheduler.
3476
-
3477
- One can check if the access key is well configured using:
3478
-
3479
- ```bash
3480
- <%=cmd%> -Ppreviewconf node browse /
3481
- ```
3482
-
3483
- This shall list the contents of the storage root of the access key.
3484
-
3485
- ### Execution
3486
-
3487
- The tool intentionally supports only a "one shot" mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method).
3488
- It needs to be run on a regular basis to create or update preview files. For that use your best
3489
- reliable scheduler. For instance use "CRON" on Linux or Task Scheduler on Windows.
3490
-
3491
- Typically, for "Access key" access, the system/transfer is `xfer`. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user `xfer`.
3492
-
3493
- Lets do a one shot test, using the configuration previously created:
3494
-
3495
- ```bash
3496
- su -s /bin/bash - xfer
3497
-
3498
- <%=cmd%> preview scan --overwrite=always
3499
- ```
3500
-
3501
- When the preview generator is first executed it will create a file: `.aspera_access_key`
3502
- in the previews folder which contains the access key used.
3503
- On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.
3504
-
3505
- ### Configuration for Execution in scheduler
3506
-
3507
- Here is an example of configuration for use with cron on Linux.
3508
- Adapt the scripts to your own preference.
3509
-
3510
- We assume here that a configuration preset was created as shown previously.
3511
-
3512
- Lets first setup a script that will be used in the scheduler and sets up the environment.
3513
-
3514
- Example of startup script `cron_<%=cmd%>`, which sets the Ruby environment and adds some timeout protection:
3515
-
3516
- ```bash
3517
- #!/bin/bash
3518
- # set a timeout protection, just in case
3519
- case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
3520
- . /etc/profile.d/rvm.sh
3521
- rvm use 2.6 --quiet
3522
- exec timeout ${tmout} <%=cmd%> "${@}"
3523
- ```
3524
-
3525
- Here the cronjob is created for user `xfer`.
3526
-
3527
- ```bash
3528
- crontab<<EOF
3529
- 0 * * * * /home/xfer/cron_<%=cmd%> preview scan --logger=syslog --display=error
3530
- 2-59 * * * * /home/xfer/cron_<%=cmd%> preview trev --logger=syslog --display=error
3531
- EOF
3532
- ```
3533
-
3534
- Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.
3535
-
3536
- ### Candidate detection for creation or update (or deletion)
3537
-
3538
- The tool generates preview files using those commands:
3539
-
3540
- * `trevents` : only recently uploaded files will be tested (transfer events)
3541
- * `events` : only recently uploaded files will be tested (file events: not working)
3542
- * `scan` : recursively scan all files under the access key&apos;s "storage root"
3543
- * `test` : test using a local file
3544
-
3545
- Once candidate are selected, once candidates are selected,
3546
- a preview is always generated if it does not exist already,
3547
- else if a preview already exist, it will be generated
3548
- using one of three values for the `overwrite` option:
3549
-
3550
- * `always` : preview is always generated, even if it already exists and is newer than original
3551
- * `never` : preview is generated only if it does not exist already
3552
- * `mtime` : preview is generated only if the original file is newer than the existing
3553
-
3554
- Deletion of preview for deleted source files: not implemented yet (TODO).
3555
-
3556
- If the `scan` or `events` detection method is used, then the option : `skip_folders` can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the `@json:` notation, example:
3557
-
3558
- ```bash
3559
- <%=cmd%> preview scan --skip-folders=@json:'["/not_here"]'
3560
- ```
3561
-
3562
- The option `folder_reset_cache` forces the node service to refresh folder contents using various methods.
3563
-
3564
- When scanning the option `value` has the same behavior as for the `node find` command.
3565
-
3566
- For instance to filter out files beginning with `._` do:
3567
-
3568
- ```bash
3569
- ... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'
3570
- ```
3571
-
3572
- ### Preview File types
3573
-
3574
- Two types of preview can be generated:
3575
-
3576
- * png: thumbnail
3577
- * mp4: video preview (only for video)
3578
-
3579
- Use option `skip_format` to skip generation of a format.
3580
-
3581
- ### Supported input Files types
3582
-
3583
- The preview generator supports rendering of those file categories:
3584
-
3585
- * image
3586
- * pdf
3587
- * plaintext
3588
- * office
3589
- * video
3590
-
3591
- To avoid generation for some categories, specify a list using option `skip_types`.
3592
-
3593
- Each category has a specific rendering method to produce the png thumbnail.
3594
-
3595
- The mp4 video preview file is only for category `video`
3596
-
3597
- File type is primarily based on file extension detected by the node API and translated info a mime type returned by the node API.
3598
-
3599
- The tool can also locally detect the mime type using gem `mimemagic`.
3600
-
3601
- ### Access to original files and preview creation
3602
-
3603
- Standard open source tools are used to create thumbnails and video previews.
3604
- Those tools require that original files are accessible in the local file system and also write generated files on the local file system.
3605
- The tool provides 2 ways to read and write files with the option: `file_access`
3606
-
3607
- If the preview generator is run on a system that has direct access to the file system, then the value `local` can be used. In this case, no transfer happen, source files are directly read from the storage, and preview files
3608
- are directly written to the storage.
3609
-
3610
- If the preview generator does not have access to files on the file system (it is remote, no mount, or is an object storage), then the original file is first downloaded, then the result is uploaded, use method `remote`.
3611
-
3612
- ## SMTP for email notifications
3613
-
3614
- Aspera CLI can send email, for that setup SMTP configuration. This is done with option `smtp`.
3615
-
3616
- The `smtp` option is a hash table (extended value) with the following fields:
3617
- <table>
3618
- <tr><th>field</th><th>default</th><th>example</th><th>description</th></tr>
3619
- <tr><td>`server`</td><td>-</td><td>smtp.gmail.com</td><td>SMTP server address</td></tr>
3620
- <tr><td>`tls`</td><td>true</td><td>false</td><td>use of TLS</td></tr>
3621
- <tr><td>`port`</td><td>587 for tls<br/>25 else</td><td>587</td><td>port for service</td></tr>
3622
- <tr><td>`domain`</td><td>domain of server</td><td>gmail.com</td><td>email domain of user</td></tr>
3623
- <tr><td>`username`</td><td>-</td><td>john@example.com</td><td>user to authenticate on SMTP server, leave empty for open auth.</td></tr>
3624
- <tr><td>`password`</td><td>-</td><td>MyP@ssword</td><td>password for above username</td></tr>
3625
- <tr><td>`from_email`</td><td>username if defined</td><td>laurent.martin.l@gmail.com</td><td>address used if received replies</td></tr>
3626
- <tr><td>`from_name`</td><td>same as email</td><td>John Wayne</td><td>display name of sender</td></tr>
3627
- </table>
3628
-
3629
- ### Example of configuration:
3630
-
3631
- ```bash
3632
- <%=cmd%> config preset set smtp_google server smtp.google.com
3633
- <%=cmd%> config preset set smtp_google username john@gmail.com
3634
- <%=cmd%> config preset set smtp_google password P@ssw0rd
3635
- ```
3636
-
3637
- or
3638
-
3639
- ```javascript
3640
- <%=cmd%> config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"P@ssw0rd"}'
3641
- ```
3642
-
3643
- or
3644
-
3645
- ```bash
3646
- <%=cmd%> config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=P@ssw0rd
3647
- ```
3648
-
3649
- Set this configuration as global default, for instance:
3650
-
3651
- ```bash
3652
- <%=cmd%> config preset set cli_default smtp @val:@preset:smtp_google
3653
- <%=cmd%> config preset set default config cli_default
3654
- ```
3655
-
3656
- ### Email templates
3657
-
3658
- Sent emails are built using a template that uses the [ERB](https://www.tutorialspoint.com/ruby/eruby.htm) syntax.
3659
-
3660
- The template is the full SMTP message, including headers.
3661
-
3662
- The following variables are defined by default:
3663
-
3664
- * from_name
3665
- * from_email
3666
- * to
3667
-
3668
- Other variables are defined depending on context.
3669
-
3670
- ### Test
3671
-
3672
- Check settings with `smtp_settings` command. Send test email with `email_test`.
3673
-
3674
- ```bash
3675
- <%=cmd%> config --smtp=@preset:smtp_google smtp
3676
- <%=cmd%> config --smtp=@preset:smtp_google email --notif-to=sample.dest@example.com
3677
- ```
3678
-
3679
- ### Notifications for transfer status
3680
-
3681
- An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
3682
-
3683
- To activate, use option `notif_to`.
3684
-
3685
- A default e-mail template is used, but it can be overridden with option `notif_template`.
3686
-
3687
- The environment provided contains the following additional variables:
3688
-
3689
- * subject
3690
- * body
3691
- * global_transfer_status
3692
- * ts
3693
-
3694
- Example of template:
3695
-
3696
- ```text
3697
- From: <%='<'%>%=from_name%> <<%='<'%>%=from_email%>>
3698
- To: <<%='<'%>%=to%>>
3699
- Subject: <%='<'%>%=subject%>
3700
-
3701
- Transfer is: <%='<'%>%=global_transfer_status%>
3702
- ```
3703
-
3704
- ## Tool: `asession`
3705
-
3706
- This gem comes with a second executable tool providing a simplified standardized interface
3707
- to start a FASP session: `asession`.
3708
-
3709
- It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a [_transfer-spec_](#transferspec) is:
3710
-
3711
- * common to Aspera Node API (HTTP POST /ops/transfer)
3712
- * common to Aspera Connect API (browser javascript startTransfer)
3713
- * easy to generate by using any third party language specific JSON library
3714
-
3715
- Hopefully, IBM integrates this diectly in `ascp`, and this tool is made redundant.
3716
-
3717
- This makes it easy to integrate with any language provided that one can spawn a sub process, write to its STDIN, read from STDOUT, generate and parse JSON.
3718
-
3719
- The tool expect one single argument: a [_transfer-spec_](#transferspec).
3720
-
3721
- If not argument is provided, it assumes a value of: `@json:@stdin:`, i.e. a JSON formatted [_transfer-spec_](#transferspec) on stdin.
3722
-
3723
- Note that if JSON is the format, one has to specify `@json:` to tell the tool to decode the hash using JSON.
3724
-
3725
- During execution, it generates all low level events, one per line, in JSON format on stdout.
3726
-
3727
- Note that there are special "extended" [_transfer-spec_](#transferspec) parameters supported by `asession`:
3728
-
3729
- * `EX_loglevel` to change log level of the tool
3730
- * `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
3731
-
3732
- Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the [`direct`](#agt_direct) transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
3733
-
3734
- ### Comparison of interfaces
3735
-
3736
- <table>
3737
- <tr><th>feature/tool</th><th>asession</th><th>ascp</th><th>FaspManager</th><th>Transfer SDK</th></tr>
3738
- <tr><td>language integration</td><td>any</td><td>any</td><td>C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/></td><td>any</td></tr>
3739
- <tr><td>additional components to ascp</td><td>Ruby<br/>Aspera</td><td>-</td><td>library<br/>(headers)</td><td>daemon</td></tr>
3740
- <tr><td>startup</td><td>JSON on stdin<br/>(standard APIs:<br/>JSON.generate<br/>Process.spawn)</td><td>command line arguments</td><td>API</td><td>daemon</td></tr>
3741
- <tr><td>events</td><td>JSON on stdout</td><td>none by default<br/>or need to open management port<br/>and proprietary text syntax</td><td>callback</td><td>callback</td></tr>
3742
- <tr><td>platforms</td><td>any with ruby and ascp</td><td>any with ascp</td><td>any with ascp</td><td>any with ascp and transferdaemon</td></tr></table>
3743
-
3744
- ### Simple session
3745
-
3746
- ```javascript
3747
- MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_demo_pass_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'
3748
-
3749
- echo "${MY_TSPEC}"|asession
3750
- ```
3751
-
3752
- ### Asynchronous commands and Persistent session
3753
-
3754
- `asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -&gt; `long_parameter`
3755
-
3756
- This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
3757
-
3758
- ```javascript
3759
- asession
3760
- {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_demo_pass_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
3761
- {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
3762
- {"type":"DONE"}
3763
- ```
3764
-
3765
- (events from FASP are not shown in above example. They would appear after each command)
3766
-
3767
- ### Example of language wrapper
3768
-
3769
- Nodejs: [https://www.npmjs.com/package/aspera](https://www.npmjs.com/package/aspera)
3770
-
3771
- ### Help
3772
-
3773
- ```bash
3774
- asession -h
3775
- <%=File.read(ENV["INCL_ASESSION"])%>
3776
- ```
3777
-
3778
- ## Hot folder
3779
-
3780
- ### Requirements
3781
-
3782
- <%=tool%> maybe used as a simple hot folder engine. A hot folder being defined as a tool that:
3783
-
3784
- * locally (or remotely) detects new files in a top folder
3785
- * send detected files to a remote (respectively, local) repository
3786
- * only sends new files, do not re-send already sent files
3787
- * optionally: sends only files that are not still "growing"
3788
- * optionally: after transfer of files, deletes or moves to an archive
3789
-
3790
- In addition: the detection should be made "continuously" or on specific time/date.
3791
-
3792
- ### Setup procedure
3793
-
3794
- The general idea is to rely on :
3795
-
3796
- * existing `ascp` features for detection and transfer
3797
- * take advantage of <%=tool%> configuration capabilities and server side knowledge
3798
- * the OS scheduler for reliability and continuous operation
3799
-
3800
- #### ascp features
3801
-
3802
- Interesting ascp features are found in its arguments: (see ascp manual):
3803
-
3804
- * `ascp` already takes care of sending only "new" files: option `-k 1,2,3`, or transfer_spec: `resume_policy`
3805
- * `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories`
3806
- * `ascp` has an option to send only files not modified since the last X seconds: `--exclude-newer-than` (--exclude-older-than)
3807
- * `--src-base` if top level folder name shall not be created on destination
3808
-
3809
- Note that:
3810
-
3811
- * <%=tool%> takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
3812
- * most, but not all native ascp arguments are available as standard transfer_spec parameters
3813
- * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the [`direct`](#agt_direct) transfer agent (not connect or node)
3814
-
3815
- #### server side and configuration
3816
-
3817
- Virtually any transfer on a "repository" on a regular basis might emulate a hot folder. Note that file detection is not based on events (inotify, etc...), but on a stateless scan on source side.
3818
-
3819
- Note: parameters may be saved in a <%=prst%> and used with `-P`.
3820
-
3821
- #### Scheduling
3822
-
3823
- Once <%=tool%> parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc... Refer to section [_Scheduling_](#_scheduling_).
3824
-
3825
- ### Example: upload folder
3826
-
3827
- ```javascript
3828
- <%=cmd%> server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'
3829
- ```
3830
-
3831
- The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).
3832
-
3833
- ## Health check and Nagios
3834
-
3835
- Most plugin provide a `health` command that will check the health status of the application. Example:
3836
-
3837
- ```bash
3838
- <%=cmd%> console health
3839
- ```
3840
-
3841
- ```output
3842
- +--------+-------------+------------+
3843
- | status | component | message |
3844
- +--------+-------------+------------+
3845
- | ok | console api | accessible |
3846
- +--------+-------------+------------+
3847
- ```
3848
-
3849
- Typically, the health check uses the REST API of the application with the following exception: the `server` plugin allows checking health by:
3850
-
3851
- * issuing a transfer to the server
3852
- * checking web app status with `asctl all:status`
3853
- * checking daemons process status
3854
-
3855
- <%=tool%> can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option `--format=nagios` :
3856
-
3857
- ```bash
3858
- <%=cmd%> server health transfer --to-folder=/Upload --format=nagios --progress=none
3859
- ```
3860
-
3861
- ```output
3862
- OK - [transfer:ok]
3863
- ```
3864
-
3865
- ```bash
3866
- <%=cmd%> server health asctlstatus --cmd_prefix='sudo ' --format=nagios
3867
- ```
3868
-
3869
- ```output
3870
- OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]
3871
- ```
3872
-
3873
- ## Ruby Module: `Aspera`
3874
-
3875
- Main components:
3876
-
3877
- * `Aspera` generic classes for REST and OAuth
3878
- * `Aspera::Fasp`: starting and monitoring transfers. It can be considered as a FASPManager class for Ruby.
3879
- * `Aspera::Cli`: <%=tool%>.
3880
-
3881
- A working example can be found in the gem, example:
3882
-
3883
- ```bash
3884
- <%=cmd%> config gem_path
3885
- ```
3886
-
3887
- ```bash
3888
- cat $(<%=cmd%> config gem_path)/../examples/transfer.rb
3889
- ```
3890
-
3891
- This sample code shows some example of use of the API as well as REST API.
3892
- Note: although nice, it's probably a good idea to use RestClient for REST.
3893
-
3894
- Example of use of the API of Aspera on Cloud:
3895
-
3896
- ```ruby
3897
- require 'aspera/aoc'
3898
-
3899
- aoc=Aspera::AoC.new(url: 'https://sedemo.ibmaspera.com',auth: :jwt, scope: 'user:all', private_key: File.read(File.expand_path('~/.aspera/<%=cmd%>/aspera_on_cloud_key')),username: 'laurent.martin.aspera@fr.ibm.com',subpath: 'api/v1')
3900
-
3901
- aoc.read('self')
3902
- ```
3903
-
3904
- <https://github.com/IBM/aspera-cli/blob/main/examples/aoc.rb>
3905
-
3906
- ## History
3907
-
3908
- When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
3909
-
3910
- There were a few pitfalls:
3911
-
3912
- * The tool was written in the aging `perl` language while most Aspera application products (but the Transfer Server) are written in `ruby`.
3913
- * The tool was only for transfers, but not able to call other products APIs
3914
-
3915
- So, it evolved into <%=tool%>:
3916
-
3917
- * portable: works on platforms supporting `ruby` (and `ascp`)
3918
- * easy to install with the `gem` utility
3919
- * supports transfers with multiple [Transfer Agents](#agents), that&apos;s why transfer parameters moved from ascp command line to [_transfer-spec_](#transferspec) (more reliable , more standard)
3920
- * `ruby` is consistent with other Aspera products
3921
-
3922
- ## Changes (Release notes)
3923
-
3924
- * <%=gemspec.version.to_s%>
3925
-
3926
- * new: command `conf plugin create`
3927
- * new: global option `plugin_folder`
3928
- * new: global option `transpose_single`
3929
- * new: simplified metadata passing for shared inbox package creation in AoC
3930
- * change: (break) command `aoc packages shared_inboxes list` replaces `aoc user shared_inboxes`
3931
- * change: (break) command `aoc user profile` replaces `aoc user info`
3932
- * change: (break) command `aoc user workspaces list` replaces `aoc user workspaces`
3933
- * change: (break) command `aoc user workspaces current` replaces `aoc workspace`
3934
- * change: (break) command `conf plugin list` replaces `conf plugins`
3935
- * change: (break) command `conf connect` simplified
3936
- * fix: #60 ascli executable was not installed by default in 4.5.0
3937
- * fix: add password hiding case in logs
3938
-
3939
- * 4.5.0
3940
-
3941
- * new: support transfer agent: [Transfer SDK](#agt_trsdk)
3942
- * new: support [http socket options](#http_options)
3943
- * new: logs hide passwords and secrets, option `log_passwords` to enable logging secrets
3944
- * new: `config vault` supports encrypted passwords, also macos keychain
3945
- * new: `config preset` command for consistency with id
3946
- * new: identifier can be provided using either option `id` or directly after the command, e.g. `delete 123` is the same as `delete --id=123`
3947
- * change: when using wss, use [ruby's CA certs](#certificates)
3948
- * change: unexpected parameter makes exit code not zero
3949
- * change: (break) options `id` and `name` cannot be specified at the same time anymore, use [positional identifer or name selection](#res_select)
3950
- * change: (break) `aoc admin res node` does not take workspace main node as default node if no `id` specified.
3951
- * change: (break): `orchestrator workflow status` requires id, and supports special id `ALL`
3952
- * fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
3953
-
3954
- * 4.4.0
3955
-
3956
- * new: `aoc packages list` add possibility to add filter with option `query`
3957
- * new: `aoc admin res xxx list` now get all items by default #50
3958
- * new: `preset` option can specify name or hash value
3959
- * new: `node` plugin accepts bearer token and access key as credential
3960
- * new: `node` option `token_type` allows using basic token in addition to aspera type.
3961
- * change: `server`: option `username` not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
3962
- * change: (break) resource `apps_new` of `aoc` replaced with `application` (more clear)
3963
-
3964
- * 4.3.0
3965
-
3966
- * new: parameter `multi_incr_udp` for option `transfer_info`: control if UDP port is incremented when multi-session is used on [`direct`](#agt_direct) transfer agent.
3967
- * new: command `aoc files node_info` to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
3968
-
3969
- * 4.2.2
3970
-
3971
- * new: `faspex package list` retrieves the whole list, not just first page
3972
- * new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem `webrick`
3973
- * new: the error "Remote host is not who we expected" displays a special remediation message
3974
- * new: `conf ascp spec` displays supported transfer spec
3975
- * new: options `notif_to` and `notif_template` to send email notifications on transfer (and other events)
3976
- * fix: space character in `faspe:` url are precent encoded if needed
3977
- * fix: `preview scan`: if file_id is unknown, ignore and continue scan
3978
- * change: for commands that potentially execute several transfers (`package recv --id=ALL`), if one transfer fails then <%=tool%> exits with code 1 (instead of zero=success)
3979
- * change: (break) option `notify` or `aoc` replaced with `notif_to` and `notif_template`
3980
-
3981
- * 4.2.1
3982
-
3983
- * new: command `faspex package recv` supports link of type: `faspe:`
3984
- * new: command `faspex package recv` supports option `recipient` to specify dropbox with leading `*`
3985
-
3986
- * 4.2.0
3987
-
3988
- * new: command `aoc remind` to receive organization membership by email
3989
- * new: in `preview` option `value` to filter out on file name
3990
- * new: `initdemo` to initialize for demo server
3991
- * new: [`direct`](#agt_direct) transfer agent options: `spawn_timeout_sec` and `spawn_delay_sec`
3992
- * fix: on Windows `conf ascp use` expects ascp.exe
3993
- * fix: (break) multi_session_threshold is Integer, not String
3994
- * fix: `conf ascp install` renames sdk folder if it already exists (leftover shared lib may make fail)
3995
- * fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
3996
- * change: (break) `aoc apiinfo` is removed, use `aoc servers` to provide the list of cloud systems
3997
- * change: (break) parameters for resume in `transfer-info` for [`direct`](#agt_direct) are now in sub-key `"resume"`
3998
-
3999
- * 4.1.0
4000
-
4001
- * fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
4002
- * new: update documentation with regard to offline and docker installation
4003
- * new: renamed command `nagios_check` to `health`
4004
- * new: agent `http_gw` now supports upload
4005
- * new: added option `sdk_url` to install SDK from local file for offline install
4006
- * new: check new gem version periodically
4007
- * new: the --fields= option, support -_fieldname_ to remove a field from default fields
4008
- * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
4009
- * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
4010
- * new: AoC a password can be provided for a public link
4011
- * new: `conf doc` take an optional parameter to go to a section
4012
- * new: initial support for Faspex 5 Beta 1
4013
-
4014
- * 4.0.0
4015
-
4016
- * now available as open source at [<%=gemspec.homepage%>](<%=gemspec.homepage%>) with general cleanup
4017
- * changed default tool name from `mlia` to `ascli`
4018
- * changed `aspera` command to `aoc`
4019
- * changed gem name from `asperalm` to `aspera-cli`
4020
- * changed module name from `Asperalm` to `Aspera`
4021
- * removed command `folder` in `preview`, merged to `scan`
4022
- * persistency files go to sub folder instead of main folder
4023
- * added possibility to install SDK: `config ascp install`
4024
-
4025
- * 0.11.8
4026
-
4027
- * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb
4028
-
4029
- * 0.11.7
4030
-
4031
- * rework on rest call error handling
4032
- * use option `display` with value `data` to remove out of extraneous information
4033
- * fixed option `lock_port` not working
4034
- * generate special icon if preview failed
4035
- * possibility to choose transfer progress bar type with option `progress`
4036
- * AoC package creation now output package id
4037
-
4038
- * 0.11.6
4039
-
4040
- * orchestrator : added more choice in auth type
4041
- * preview: cleanup in generator (removed and renamed parameters)
4042
- * preview: better documentation
4043
- * preview: animated thumbnails for video (option: `video_png_conv=animated`)
4044
- * preview: new event trigger: `trevents` (`events` seems broken)
4045
- * preview: unique tmp folder to avoid clash of multiple instances
4046
- * repo: added template for secrets used for testing
4047
-
4048
- * 0.11.5
4049
-
4050
- * added option `default_ports` for AoC (see manual)
4051
- * allow bulk delete in `aspera files` with option `bulk=yes`
4052
- * fix getting connect versions
4053
- * added section for Aix
4054
- * support all ciphers for [`direct`](#agt_direct) agent (including gcm, etc..)
4055
- * added transfer spec param `apply_local_docroot` for [`direct`](#agt_direct)
4056
-
4057
- * 0.11.4
4058
-
4059
- * possibility to give shared inbox name when sending a package (else use id and type)
4060
-
4061
- * 0.11.3
4062
-
4063
- * minor fixes on multi-session: avoid exception on progress bar
4064
-
4065
- * 0.11.2
4066
-
4067
- * fixes on multi-session: progress bat and transfer spec param for "direct"
4068
-
4069
- * 0.11.1
4070
-
4071
- * enhanced short_link creation commands (see examples)
4072
-
4073
- * 0.11
4074
-
4075
- * add transfer spec option (agent `direct` only) to provide file list directly to ascp: `EX_file_list`.
4076
-
4077
- * 0.10.18
4078
-
4079
- * new option in. `server` : `ssh_options`
4080
-
4081
- * 0.10.17
4082
-
4083
- * fixed problem on `server` for option `ssh_keys`, now accepts both single value and list.
4084
- * new modifier: `@list:<separator>val1<separator>...`
4085
-
4086
- * 0.10.16
4087
-
4088
- * added list of shared inboxes in workspace (or global), use `--query=@json:'{}'`
4089
-
4090
- * 0.10.15
4091
-
4092
- * in case of command line error, display the error cause first, and non-parsed argument second
4093
- * AoC : Activity / Analytics
4094
-
4095
- * 0.10.14
4096
-
4097
- * added missing bss plugin
4098
-
4099
- * 0.10.13
4100
-
4101
- * added Faspex5 (use option `value` to give API arguments)
4102
-
4103
- * 0.10.12
4104
-
4105
- * added support for AoC node registration keys
4106
- * replaced option : `local_resume` with `transfer_info` for agent [`direct`](#agt_direct)
4107
- * Transfer agent is no more a Singleton instance, but only one is used in CLI
4108
- * `@incps` : new extended value modifier
4109
- * ATS: no more provides access keys secrets: now user must provide it
4110
- * begin work on "aoc" transfer agent
4111
-
4112
- * 0.10.11
4113
-
4114
- * minor refactor and fixes
4115
-
4116
- * 0.10.10
4117
-
4118
- * fix on documentation
4119
-
4120
- * 0.10.9.1
4121
-
4122
- * add total number of items for AoC resource list
4123
- * better gem version dependency (and fixes to support Ruby 2.0.0)
4124
- * removed aoc search_nodes
4125
-
4126
- * 0.10.8
4127
-
4128
- * removed option: `fasp_proxy`, use pseudo transfer spec parameter: `EX_fasp_proxy_url`
4129
- * removed option: `http_proxy`, use pseudo transfer spec parameter: `EX_http_proxy_url`
4130
- * several other changes..
4131
-
4132
- * 0.10.7
4133
-
4134
- * fix: <%=cmd%> fails when username cannot be computed on Linux.
4135
-
4136
- * 0.10.6
4137
-
4138
- * FaspManager: transfer spec `authentication` no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
4139
- * gem version requirements made more open
4140
-
4141
- * 0.10.5
4142
-
4143
- * fix faspex package receive command not working
4144
-
4145
- * 0.10.4
4146
-
4147
- * new options for AoC : `secrets`
4148
- * ACLI-533 temp file list folder to use file lists is set by default, and used by asession
4149
-
4150
- * 0.10.3
4151
-
4152
- * included user name in oauth bearer token cache for AoC when JWT is used.
4153
-
4154
- * 0.10.2
4155
-
4156
- * updated `search_nodes` to be more generic, so it can search not only on access key, but also other queries.
4157
- * added doc for "cargo" like actions
4158
- * added doc for multi-session
4159
-
4160
- * 0.10.1
4161
-
4162
- * AoC and node v4 "browse" works now on non-folder items: file, link
4163
- * initial support for AoC automation (do not use yet)
4164
-
4165
- * 0.10
4166
-
4167
- * support for transfer using IBM Cloud Object Storage
4168
- * improved `find` action using arbitrary expressions
4169
-
4170
- * 0.9.36
4171
-
4172
- * added option to specify file pair lists
4173
-
4174
- * 0.9.35
4175
-
4176
- * updated plugin `preview` , changed parameter names, added documentation
4177
- * fix in `ats` plugin : instance id needed in request header
4178
-
4179
- * 0.9.34
4180
-
4181
- * parser "@preset" can be used again in option "transfer_info"
4182
- * some documentation re-organizing
4183
-
4184
- * 0.9.33
4185
-
4186
- * new command to display basic token of node
4187
- * new command to display bearer token of node in AoC
4188
- * the --fields= option, support +_fieldname_ to add a field to default fields
4189
- * many small changes
4190
-
4191
- * 0.9.32
4192
-
4193
- * all Faspex public links are now supported
4194
- * removed faspex operation recv_publink
4195
- * replaced with option `link` (consistent with AoC)
4196
-
4197
- * 0.9.31
4198
-
4199
- * added more support for public link: receive and send package, to user or dropbox and files view.
4200
- * delete expired file lists
4201
- * changed text table gem from text-table to terminal-table because it supports multiline values
4202
-
4203
- * 0.9.27
4204
-
4205
- * basic email support with SMTP
4206
- * basic proxy auto config support
4207
-
4208
- * 0.9.26
4209
-
4210
- * table display with --fields=ALL now includes all column names from all lines, not only first one
4211
- * unprocessed argument shows error even if there is an error beforehand
4212
-
4213
- * 0.9.25
4214
-
4215
- * the option `value` of command `find`, to filter on name, is not optional
4216
- * `find` now also reports all types (file, folder, link)
4217
- * `find` now is able to report all fields (type, size, etc...)
4218
-
4219
- * 0.9.24
4220
-
4221
- * fix bug where AoC node to node transfer did not work
4222
- * fix bug on error if ED25519 private key is defined in .ssh
4223
-
4224
- * 0.9.23
4225
-
4226
- * defined REST error handlers, more error conditions detected
4227
- * commands to select specific ascp location
4228
-
4229
- * 0.9.21
4230
-
4231
- * supports simplified wizard using global client
4232
- * only ascp binary is required, other SDK (keys) files are now generated
4233
-
4234
- * 0.9.20
4235
-
4236
- * improved wizard (prepare for AoC global client id)
4237
- * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
4238
- * removed outdated pictures from this doc
4239
-
4240
- * 0.9.19
4241
-
4242
- * added command aspera bearer --scope=xx
4243
-
4244
- * 0.9.18
4245
-
4246
- * enhanced aspera admin events to support query
4247
-
4248
- * 0.9.16
4249
-
4250
- * AoC transfers are now reported in activity app
4251
- * new interface for Rest class authentication (keep backward compatibility)
4252
-
4253
- * 0.9.15
4254
-
4255
- * new feature: "find" command in aspera files
4256
- * sample code for transfer API
4257
-
4258
- * 0.9.12
4259
-
4260
- * add nagios commands
4261
- * support of ATS for IBM Cloud, removed old version based on aspera id
4262
-
4263
- * 0.9.11
4264
-
4265
- * Breaking change: @stdin is now @stdin:
4266
- * support of ATS for IBM Cloud, removed old version based on aspera id
4267
-
4268
-
4269
- * 0.9.10
4270
-
4271
- * Breaking change: parameter transfer-node becomes more generic: transfer-info
4272
- * Display SaaS storage usage with command: aspera admin res node --id=nn info
4273
- * cleaner way of specifying source file list for transfers
4274
- * Breaking change: replaced download_mode option with http_download action
4275
-
4276
- * 0.9.9
4277
-
4278
- * Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
4279
- * Now supports "cargo" for Aspera on Cloud (automatic package download)
4280
-
4281
- * 0.9.8
4282
-
4283
- * Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
4284
- * AoC: share to share transfer with command "transfer"
4285
-
4286
- * 0.9.7
4287
-
4288
- * homogeneous [_transfer-spec_](#transferspec) for `node` and [`direct`](#agt_direct) transfer agents
4289
- * preview persistency goes to unique file by default
4290
- * catch mxf extension in preview as video
4291
- * Faspex: possibility to download all packages by specifying id=ALL
4292
- * Faspex: to come: cargo-like function to download only new packages with id=NEW
4293
-
4294
- * 0.9.6
4295
-
4296
- * Breaking change: `@param:`is now `@preset:` and is generic
4297
- * AoC: added command to display current workspace information
4298
-
4299
- * 0.9.5
4300
-
4301
- * new parameter: new_user_option used to choose between public_link and invite of external users.
4302
- * fixed bug in wizard, and wizard uses now product detection
4303
-
4304
- * 0.9.4
4305
-
4306
- * Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
4307
- * AoC Package send supports external users
4308
- * new command to export AoC config to Aspera CLI config
4309
-
4310
- * 0.9.3
4311
-
4312
- * REST error message show host and code
4313
- * option for quiet display
4314
- * modified transfer interface and allow token re-generation on error
4315
- * async add admin command
4316
- * async add db parameters
4317
- * Breaking change: new option "sources" to specify files to transfer
4318
-
4319
- * 0.9.2
4320
-
4321
- * Breaking change: changed AoC package creation to match API, see AoC section
4322
-
4323
- * 0.9.1
4324
-
4325
- * Breaking change: changed faspex package creation to match API, see Faspex section
4326
-
4327
- * 0.9
4328
-
4329
- * Renamed the CLI from aslmcli to <%=tool%>
4330
- * Automatic rename and conversion of former config folder from aslmcli to <%=tool%>
4331
-
4332
- * 0.7.6
4333
-
4334
- * add "sync" plugin
4335
-
4336
- * 0.7
4337
-
4338
- * Breaking change: AoC package recv take option if for package instead of argument.
4339
- * Breaking change: Rest class and Oauth class changed init parameters
4340
- * AoC: receive package from public link
4341
- * select by col value on output
4342
- * added rename (AoC, node)
4343
-
4344
- * 0.6.19
4345
-
4346
- * change: (break) ats server list provisioned &rarr; ats cluster list
4347
- * change: (break) ats server list clouds &rarr; ats cluster clouds
4348
- * change: (break) ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
4349
- * change: (break) ats server id xxx &rarr; ats cluster show --id=xxx
4350
- * change: (break) ats subscriptions &rarr; ats credential subscriptions
4351
- * change: (break) ats api_key repository list &rarr; ats credential cache list
4352
- * change: (break) ats api_key list &rarr; ats credential list
4353
- * change: (break) ats access_key id xxx &rarr; ats access_key --id=xxx
4354
-
4355
- * 0.6.18
4356
-
4357
- * some commands take now --id option instead of id command.
4358
-
4359
- * 0.6.15
4360
-
4361
- * Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in <%=prst%> "default".
4362
-
4363
- ## BUGS, FEATURES, CONTRIBUTION
4364
-
4365
- For issues or feature requests use the Github repository and issues.
4366
-
4367
- You can also contribute to this open source project.
4368
-
4369
- One can also [create one's own plugin](#createownplugin).
4370
-
4371
- ### Only one value for any option
4372
-
4373
- Some commands and sub commands may ask for the same option name.
4374
- Currently, since option definition is position independent (last one wins), it is not possible
4375
- to give an option to a command and the same option with different value to a sub command.
4376
-
4377
- For instance, if an entity is identified by the option `id` but later on the command line another `id` option is required, then the later will override the earlier one, and both entity will use the same id.
4378
- As a solution, use the position specific notation for selection, i.e. provide the identified just after command and do not use option `id`.
4379
-
4380
- This happens typically for the `node` sub command, e.g. identify the node by name instead of id.
4381
-
4382
- ### ED255519 key not supported
4383
-
4384
- ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.
4385
-
4386
- Without this deactivation, if such key was present the following error was generated:
4387
-
4388
- ```output
4389
- OpenSSH keys only supported if ED25519 is available
4390
- ```
4391
-
4392
- Which meant that you do not have ruby support for ED25519 SSH keys.
4393
- You may either install the suggested Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue.
4394
-
4395
- ### Error "Remote host is not who we expected"
4396
-
4397
- Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). `aspera.conf` supports a single fingerprint.
4398
-
4399
- Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
4400
-
4401
- ```bash
4402
- --ts=@json:'{"sshfp":null}'
4403
- ```
4404
-
4405
- Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`.
4406
-
4407
- References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server).
4408
-
4409
- ### Miscellaneous
4410
-
4411
- * remove rest and oauth classes and use ruby standard gems:
4412
-
4413
- * oauth
4414
- * <https://github.com/rest-client/rest-client>
4415
-
4416
- * use Thor or any standard Ruby CLI manager
4417
-
4418
- * provide metadata in packages
4419
-
4420
- * deliveries to dropboxes
4421
-
4422
- * Going through proxy: use env var http_proxy and https_proxy, no_proxy
4423
-
4424
- * easier use with <https://github.com/pmq20/ruby-packer>