aspera-cli 4.12.0 → 4.14.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (80) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/CHANGELOG.md +45 -5
  4. data/CONTRIBUTING.md +113 -22
  5. data/README.md +1289 -754
  6. data/bin/ascli +3 -3
  7. data/examples/dascli +1 -1
  8. data/examples/rubyc +24 -0
  9. data/lib/aspera/aoc.rb +63 -74
  10. data/lib/aspera/ascmd.rb +5 -3
  11. data/lib/aspera/cli/basic_auth_plugin.rb +6 -6
  12. data/lib/aspera/cli/extended_value.rb +24 -37
  13. data/lib/aspera/cli/formatter.rb +23 -25
  14. data/lib/aspera/cli/info.rb +2 -4
  15. data/lib/aspera/cli/main.rb +27 -27
  16. data/lib/aspera/cli/manager.rb +143 -120
  17. data/lib/aspera/cli/plugin.rb +88 -43
  18. data/lib/aspera/cli/plugins/alee.rb +2 -2
  19. data/lib/aspera/cli/plugins/aoc.rb +235 -104
  20. data/lib/aspera/cli/plugins/ats.rb +16 -18
  21. data/lib/aspera/cli/plugins/bss.rb +3 -3
  22. data/lib/aspera/cli/plugins/config.rb +190 -373
  23. data/lib/aspera/cli/plugins/console.rb +4 -6
  24. data/lib/aspera/cli/plugins/cos.rb +12 -13
  25. data/lib/aspera/cli/plugins/faspex.rb +21 -21
  26. data/lib/aspera/cli/plugins/faspex5.rb +399 -150
  27. data/lib/aspera/cli/plugins/node.rb +260 -174
  28. data/lib/aspera/cli/plugins/orchestrator.rb +15 -18
  29. data/lib/aspera/cli/plugins/preview.rb +40 -62
  30. data/lib/aspera/cli/plugins/server.rb +33 -16
  31. data/lib/aspera/cli/plugins/shares.rb +24 -33
  32. data/lib/aspera/cli/plugins/sync.rb +6 -6
  33. data/lib/aspera/cli/transfer_agent.rb +47 -30
  34. data/lib/aspera/cli/version.rb +2 -1
  35. data/lib/aspera/colors.rb +9 -7
  36. data/lib/aspera/command_line_builder.rb +2 -1
  37. data/lib/aspera/cos_node.rb +1 -1
  38. data/lib/aspera/data/6 +0 -0
  39. data/lib/aspera/environment.rb +7 -3
  40. data/lib/aspera/fasp/agent_connect.rb +6 -1
  41. data/lib/aspera/fasp/agent_direct.rb +17 -17
  42. data/lib/aspera/fasp/agent_httpgw.rb +138 -60
  43. data/lib/aspera/fasp/agent_node.rb +14 -4
  44. data/lib/aspera/fasp/agent_trsdk.rb +2 -0
  45. data/lib/aspera/fasp/error_info.rb +2 -0
  46. data/lib/aspera/fasp/installation.rb +19 -19
  47. data/lib/aspera/fasp/parameters.rb +29 -20
  48. data/lib/aspera/fasp/parameters.yaml +5 -2
  49. data/lib/aspera/fasp/resume_policy.rb +3 -3
  50. data/lib/aspera/fasp/transfer_spec.rb +8 -5
  51. data/lib/aspera/fasp/uri.rb +23 -21
  52. data/lib/aspera/faspex_gw.rb +1 -0
  53. data/lib/aspera/faspex_postproc.rb +3 -3
  54. data/lib/aspera/hash_ext.rb +12 -2
  55. data/lib/aspera/keychain/macos_security.rb +13 -13
  56. data/lib/aspera/log.rb +1 -0
  57. data/lib/aspera/node.rb +73 -84
  58. data/lib/aspera/oauth.rb +4 -3
  59. data/lib/aspera/persistency_action_once.rb +1 -1
  60. data/lib/aspera/preview/file_types.rb +8 -6
  61. data/lib/aspera/preview/generator.rb +23 -11
  62. data/lib/aspera/preview/options.rb +3 -2
  63. data/lib/aspera/preview/terminal.rb +80 -0
  64. data/lib/aspera/preview/utils.rb +11 -11
  65. data/lib/aspera/proxy_auto_config.js +2 -2
  66. data/lib/aspera/rest.rb +42 -4
  67. data/lib/aspera/rest_call_error.rb +3 -1
  68. data/lib/aspera/secret_hider.rb +10 -5
  69. data/lib/aspera/ssh.rb +1 -1
  70. data/lib/aspera/sync.rb +41 -33
  71. data/lib/aspera/web_server_simple.rb +22 -18
  72. data.tar.gz.sig +0 -0
  73. metadata +40 -48
  74. metadata.gz.sig +0 -0
  75. data/docs/test_env.conf +0 -179
  76. data/examples/aoc.rb +0 -30
  77. data/examples/faspex4.rb +0 -94
  78. data/examples/node.rb +0 -96
  79. data/examples/server.rb +0 -93
  80. data/lib/aspera/data/7 +0 -0
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e7b048647bb937cbea1f3bd8ef7e010c18b97456aa00b45ad8be45656472ae9f
4
- data.tar.gz: c0f02324d8916a1ca5247a313da624e8365a57c438aefc58cc1301fd5ca5e0ff
3
+ metadata.gz: 98bd3e1d6ade2f96ff4ddbf4ae825e1926f56990b6552ff22aab6007546fd32e
4
+ data.tar.gz: edab64f8c8a7108a4614f4f5b7494ddb73611587c54bf6a4c270385303d9d53b
5
5
  SHA512:
6
- metadata.gz: fca3097df5295eaf1f3fa67efe04c7d1226c5c050d01f3e3c7319210967addfa01d98fd7a7b67024cae0a085f56b0357cf4b9b7ee4eb7aedd027b7996dc59356
7
- data.tar.gz: c45dc8053b73c26b7f0eaac877d35118aad007974ae61cb6b3e19aac373e8713f682513161e1af5250263bf461a03aa71846e4149fed411e81c771a2943c8e57
6
+ metadata.gz: 26c290fd4cd0d6d2a564b8e8dacaab4f584a0548c096999870b43dc755128faf23db58105ccbc16f424aac5384e79a51d07eada5e7a82e66b3d76abec69ad8e7
7
+ data.tar.gz: 1ee2fe285c913db6851ac97198d73c7159972c35c3c30e2b57342d226d0387e4940df19b7bd737769bfac3b2ab74fec3813d41a1f70c2d39e5202c20c35b8369
checksums.yaml.gz.sig CHANGED
Binary file
data/CHANGELOG.md CHANGED
@@ -1,5 +1,45 @@
1
1
  # Changes (Release notes)
2
2
 
3
+ * 4.14.0
4
+
5
+ * new: server: option `passphrase` for simpler command line (#114)
6
+ * new: percent selector for entities identifier
7
+ * new: faspex5: shared inbox and workgroup membership management
8
+ * new: faspex5: invite external user to shared inbox
9
+ * new: faspex5: package list and receive from workgroups and shared inboxes
10
+ * new: config: Command `ascp info` shows default transfer spec.
11
+ * new: httpgw: synchronous and asynchronous upload modes
12
+ * new: node: command `bandwidth_average` to get average bandwidth of node, per periods
13
+ * fix: option `ts`: deep add and remove of keys. (#117)
14
+ * fix: faspex5: user lookup for `packages send` shall be exact match (#120)
15
+ * fix: direct: if transfer spec contains "paths" and elements with "destination", but first element has only "source", then destinations were ignored. Now "destination" all or none is enforced.
16
+ * change: using `aoc files` or node gen4 operations (`browse`, `delete`) on a link will follow the link only if path ends with /
17
+ * change: shares: command `repository` is changed to `files` for consistency with aoc and upcoming faspex5, but is still available as alias
18
+ * change: aoc: better handling of shared links
19
+ * change: global: option `value` is deprecated. Use positional parameter for creation data and option `query` for list/delete operations.
20
+ * change: config: remove deprecated command: `export_to_cli`
21
+ * change: config: removed all legacy preset command, newer command `preset` shall be used now.
22
+ * change: config: SDK is now installed in $HOME/.aspera/sdk instead of $HOME/.aspera/ascli/sdk
23
+ * change(break): aoc/node: Simplification: gen4 operations: show modify permission thumbnail are now directly under node gen 4 command. Command `file` is suppressed. Option `path` is suppressed. The default expected argument is a path. To provide a file id, use selector syntax: %id:_file_id_
24
+ * change(break): node: option `token_type` is removed, as starting with HSTS 4.3 basic token is only allowed with access keys, so use gen4 operations: `acc do self`
25
+
26
+ * 4.13.0
27
+
28
+ * new: preview: option `reencode_ffmpeg` allows overriding all re-encoding options
29
+ * new: faspex5: package delete (#107)
30
+ * new: faspex5: package recv for inboxes and regular users (#108)
31
+ * new: faspex5: smtp management
32
+ * new: faspex5: use public link for authorization of package download, using option `link`
33
+ * new: faspex5: list content of package, and allow partial download of package
34
+ * new: faspex5: list packages support multiple pages and items limitations (`max` and `pmax`)
35
+ * new: aoc: files operations with workspace-less user (#109)
36
+ * new: node: async with gen3 token (#110)
37
+ * new: node: display of preview of file in terminal for access keys
38
+ * change: option `transfer_info` is now cumulative, setting several times merge values
39
+ * change(deprecation): Removed support of Ruby 2.4 and 2.5 : too old, no security update since a long time. If you need older ruby version use older gem version.
40
+ * fix: cos: do not use refresh token when not supported
41
+ * fix: container: SDK installed in other folder than `ascli` (#106)
42
+
3
43
  * 4.12.0
4
44
 
5
45
  * new: docker: build image from official gem version, possibility to deploy beta as well
@@ -172,7 +212,7 @@
172
212
  * new: agent `http_gw` now supports upload
173
213
  * new: added option `sdk_url` to install SDK from local file for offline install
174
214
  * new: check new gem version periodically
175
- * new: the --fields= option, support -_fieldname_ to remove a field from default fields
215
+ * new: the --fields= option, support -_field_name_ to remove a field from default fields
176
216
  * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
177
217
  * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
178
218
  * new: AoC a password can be provided for a public link
@@ -192,7 +232,7 @@
192
232
 
193
233
  * 0.11.8
194
234
 
195
- * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb)
235
+ * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using `Xvfb`)
196
236
 
197
237
  * 0.11.7
198
238
 
@@ -313,7 +353,7 @@
313
353
  * 0.10.4
314
354
 
315
355
  * new options for AoC : `secrets`
316
- * ACLI-533 temp file list folder to use file lists is set by default, and used by `asession`
356
+ * `ACLI-533` temp file list folder to use file lists is set by default, and used by `asession`
317
357
 
318
358
  * 0.10.3
319
359
 
@@ -353,13 +393,13 @@
353
393
 
354
394
  * new command to display basic token of node
355
395
  * new command to display bearer token of node in AoC
356
- * the --fields= option, support +_fieldname_ to add a field to default fields
396
+ * the --fields= option, support +_field_name_ to add a field to default fields
357
397
  * many small changes
358
398
 
359
399
  * 0.9.32
360
400
 
361
401
  * all Faspex public links are now supported
362
- * removed faspex operation recv_publink
402
+ * removed faspex operation `recv_publink`
363
403
  * replaced with option `link` (consistent with AoC)
364
404
 
365
405
  * 0.9.31
data/CONTRIBUTING.md CHANGED
@@ -1,5 +1,5 @@
1
1
  # Contributing
2
-
2
+ <!-- cspell:words passin GEMVERS -->
3
3
  ## Reporting Issues and Vulnerabilities
4
4
 
5
5
  You can report issues at <https://github.com/IBM/aspera-cli/issues>
@@ -34,6 +34,20 @@ If you want to contribute, please:
34
34
  - Send a pull request on GitHub.
35
35
  - run Rubocop to comply for coding standards
36
36
 
37
+ ## Architecture
38
+
39
+ A list of classes are provided in <docs/ml.png>
40
+
41
+ Architecture:
42
+
43
+ ![Architecture](docs/architecture.png)
44
+
45
+ The entry point is lib/aspera/cli/main.rb
46
+
47
+ Plugins are located in lib/aspera/cli/plugins
48
+
49
+ Transfer agents: lib/aspera/fasp
50
+
37
51
  ## Running Tests
38
52
 
39
53
  First, a testing environment must be created:
@@ -73,51 +87,111 @@ It is also possible to build a non-signed version for development purpose: `make
73
87
 
74
88
  ### Gem Signature
75
89
 
76
- Refer to: <https://guides.rubygems.org/security/>
90
+ Refer to:
77
91
 
78
- The gem is signed with the public certificate found in `certs` and a secret key (kept secret by maintainer).
92
+ - <https://guides.rubygems.org/security/>
93
+ - <https://ruby-doc.org/current/stdlibs/rubygems/Gem/Security.html>
94
+ - `gem cert --help`
79
95
 
80
- To build the signed gem:
96
+ Then procedure is as follows:
81
97
 
82
- ```bash
83
- SIGNING_KEY=/path/to/signing_key.pem make
84
- ```
98
+ - The maintainer creates the initial certificate and a private key:
85
99
 
86
- The user can activate gem signature verification on installation:
100
+ ```bash
101
+ cd /path/to/vault
102
+ gem cert --build maintainer@example.com
103
+ ```
87
104
 
88
- - Add the certificate to gem trusted certificates:
105
+ > **Note:** the email must match the field `spec.email` in `aspera-cli.gemspec`
89
106
 
90
- ```bash
91
- curl https://raw.githubusercontent.com/IBM/aspera-cli/main/certs/aspera-cli-public-cert.pem -so aspera-cli-certificate.pem
92
- gem cert --add aspera-cli-certificate.pem
93
- rm aspera-cli-certificate.pem
94
- ```
107
+ This creates two files in folder `/path/to/vault` (e.g. $HOME/.ssh):
108
+
109
+ - `gem-private_key.pem` : This file shall be kept secret in a vault.
110
+ - `gem-public_cert.pem` : This file is copied to a public place, here in folder `certs`
95
111
 
96
- - Install the gem with `HighSecurity` or `MediumSecurity`: this will succeed only of the gem is trusted.
112
+ > **Note:** Alternatively, use an existing key or generate one, and then `make new-cert`
97
113
 
98
- ```bash
99
- gem install -P HighSecurity aspera-cli
100
- ```
114
+ - The maintainer build the signed gem.
115
+
116
+ The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
117
+
118
+ To build the signed gem:
119
+
120
+ ```bash
121
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make
122
+ ```
123
+
124
+ - The user can activate gem signature verification on installation:
125
+
126
+ Add the certificate to gem trusted certificates:
127
+
128
+ ```bash
129
+ curl https://raw.githubusercontent.com/IBM/aspera-cli/main/certs/aspera-cli-public-cert.pem -so aspera-cli-certificate.pem
130
+ gem cert --add aspera-cli-certificate.pem
131
+ rm aspera-cli-certificate.pem
132
+ ```
133
+
134
+ - The user installs the gem with `HighSecurity` or `MediumSecurity`: this will succeed only of the gem is trusted.
135
+
136
+ ```bash
137
+ gem install -P HighSecurity aspera-cli
138
+ ```
139
+
140
+ - The maintainer can renew the certificate when it is expired using the same private key:
141
+
142
+ ```bash
143
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make update-cert
144
+ ```
145
+
146
+ Alternatively, to generate a new certificate with the same key:
147
+
148
+ ```bash
149
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make new-cert
150
+ ```
151
+
152
+ - Show the current certificate contents
153
+
154
+ ```bash
155
+ make show-cert
156
+ ```
157
+
158
+ > Note: to provide a passphrase add argument: `-passin pass:_value_` to `openssl`
159
+
160
+ - Check that the signing key is the same as used to sign the certificate:
161
+
162
+ ```bash
163
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make check-cert-key
164
+ ```
101
165
 
102
166
  ## Docker image build
103
167
 
104
- The default docker image build installs the official gem version directly from <rubygems.org>.
168
+ The Dockerfile template allows customizing the optional copy of gem file , versus install from rubygems,org as well as customizing the retrieval of the SDK.
169
+
170
+ ### Default image build
105
171
 
106
- By default it will build the image for the version in the current repository:
172
+ Build the image:
107
173
 
108
174
  ```bash
109
175
  make docker
110
176
  ```
111
177
 
178
+ This does the following:
179
+
180
+ - Install the official gem version directly from <rubygems.org>.
181
+ - Build the image for the version number in the current repository
182
+ - creates tags for both the version and `latest`
183
+
112
184
  > **Note:** This target creates the `Dockerfile` from an `ERB` (embedded Ruby) template (Makefile target `dockerfile`).
113
185
  A template is used as it allows some level of customization to tell where to take the gem from, as as for the SDK file.
114
186
 
115
- Then, to push to the image registry:
187
+ Then, to push to the image registry (both tags: version and `latest`):
116
188
 
117
189
  ```bash
118
190
  make dpush
119
191
  ```
120
192
 
193
+ ### Specific version image build
194
+
121
195
  It is possible to build a specific version by setting envvar `GEMVERS` and `make` with option `-e`:
122
196
 
123
197
  ```bash
@@ -125,14 +199,31 @@ GEMVERS=4.11.0 make -e docker
125
199
  GEMVERS=4.11.0 make -e dpush
126
200
  ```
127
201
 
202
+ ### Development version image build
203
+
128
204
  To build/push a beta/development container:
205
+ it does not create the `latest` tag, it uses the gem file generated locally.
129
206
 
130
207
  ```bash
131
208
  make dockerbeta
132
209
  make dpushversion
133
210
  ```
134
211
 
135
- The Dockerfile template allows customizing the optional copy of gem file , versus install from rubygems,org as well as customizing the retrieval of the SDK.
212
+ ## Single executable build
213
+
214
+ It is possible to build `ascli` as a single native executable using `rubyc` (gem `ruby-packer`).
215
+
216
+ ```bash
217
+ make single
218
+ ```
219
+
220
+ This is based on tool `rubyc` (gem `ruby-packer`): either place the executable in folder `tmp`, or the architecture-specific version will be automatically downloaded.
221
+
222
+ > **Note:** Perl Findbin is needed, install on Linux with:
223
+
224
+ ```bash
225
+ sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
226
+ ```
136
227
 
137
228
  ## Long Term Implementation and delivery improvements
138
229