aspera-cli 4.12.0 → 4.14.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/CHANGELOG.md +45 -5
  4. data/CONTRIBUTING.md +113 -22
  5. data/README.md +1289 -754
  6. data/bin/ascli +3 -3
  7. data/examples/dascli +1 -1
  8. data/examples/rubyc +24 -0
  9. data/lib/aspera/aoc.rb +63 -74
  10. data/lib/aspera/ascmd.rb +5 -3
  11. data/lib/aspera/cli/basic_auth_plugin.rb +6 -6
  12. data/lib/aspera/cli/extended_value.rb +24 -37
  13. data/lib/aspera/cli/formatter.rb +23 -25
  14. data/lib/aspera/cli/info.rb +2 -4
  15. data/lib/aspera/cli/main.rb +27 -27
  16. data/lib/aspera/cli/manager.rb +143 -120
  17. data/lib/aspera/cli/plugin.rb +88 -43
  18. data/lib/aspera/cli/plugins/alee.rb +2 -2
  19. data/lib/aspera/cli/plugins/aoc.rb +235 -104
  20. data/lib/aspera/cli/plugins/ats.rb +16 -18
  21. data/lib/aspera/cli/plugins/bss.rb +3 -3
  22. data/lib/aspera/cli/plugins/config.rb +190 -373
  23. data/lib/aspera/cli/plugins/console.rb +4 -6
  24. data/lib/aspera/cli/plugins/cos.rb +12 -13
  25. data/lib/aspera/cli/plugins/faspex.rb +21 -21
  26. data/lib/aspera/cli/plugins/faspex5.rb +399 -150
  27. data/lib/aspera/cli/plugins/node.rb +260 -174
  28. data/lib/aspera/cli/plugins/orchestrator.rb +15 -18
  29. data/lib/aspera/cli/plugins/preview.rb +40 -62
  30. data/lib/aspera/cli/plugins/server.rb +33 -16
  31. data/lib/aspera/cli/plugins/shares.rb +24 -33
  32. data/lib/aspera/cli/plugins/sync.rb +6 -6
  33. data/lib/aspera/cli/transfer_agent.rb +47 -30
  34. data/lib/aspera/cli/version.rb +2 -1
  35. data/lib/aspera/colors.rb +9 -7
  36. data/lib/aspera/command_line_builder.rb +2 -1
  37. data/lib/aspera/cos_node.rb +1 -1
  38. data/lib/aspera/data/6 +0 -0
  39. data/lib/aspera/environment.rb +7 -3
  40. data/lib/aspera/fasp/agent_connect.rb +6 -1
  41. data/lib/aspera/fasp/agent_direct.rb +17 -17
  42. data/lib/aspera/fasp/agent_httpgw.rb +138 -60
  43. data/lib/aspera/fasp/agent_node.rb +14 -4
  44. data/lib/aspera/fasp/agent_trsdk.rb +2 -0
  45. data/lib/aspera/fasp/error_info.rb +2 -0
  46. data/lib/aspera/fasp/installation.rb +19 -19
  47. data/lib/aspera/fasp/parameters.rb +29 -20
  48. data/lib/aspera/fasp/parameters.yaml +5 -2
  49. data/lib/aspera/fasp/resume_policy.rb +3 -3
  50. data/lib/aspera/fasp/transfer_spec.rb +8 -5
  51. data/lib/aspera/fasp/uri.rb +23 -21
  52. data/lib/aspera/faspex_gw.rb +1 -0
  53. data/lib/aspera/faspex_postproc.rb +3 -3
  54. data/lib/aspera/hash_ext.rb +12 -2
  55. data/lib/aspera/keychain/macos_security.rb +13 -13
  56. data/lib/aspera/log.rb +1 -0
  57. data/lib/aspera/node.rb +73 -84
  58. data/lib/aspera/oauth.rb +4 -3
  59. data/lib/aspera/persistency_action_once.rb +1 -1
  60. data/lib/aspera/preview/file_types.rb +8 -6
  61. data/lib/aspera/preview/generator.rb +23 -11
  62. data/lib/aspera/preview/options.rb +3 -2
  63. data/lib/aspera/preview/terminal.rb +80 -0
  64. data/lib/aspera/preview/utils.rb +11 -11
  65. data/lib/aspera/proxy_auto_config.js +2 -2
  66. data/lib/aspera/rest.rb +42 -4
  67. data/lib/aspera/rest_call_error.rb +3 -1
  68. data/lib/aspera/secret_hider.rb +10 -5
  69. data/lib/aspera/ssh.rb +1 -1
  70. data/lib/aspera/sync.rb +41 -33
  71. data/lib/aspera/web_server_simple.rb +22 -18
  72. data.tar.gz.sig +0 -0
  73. metadata +40 -48
  74. metadata.gz.sig +0 -0
  75. data/docs/test_env.conf +0 -179
  76. data/examples/aoc.rb +0 -30
  77. data/examples/faspex4.rb +0 -94
  78. data/examples/node.rb +0 -96
  79. data/examples/server.rb +0 -93
  80. data/lib/aspera/data/7 +0 -0
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e7b048647bb937cbea1f3bd8ef7e010c18b97456aa00b45ad8be45656472ae9f
4
- data.tar.gz: c0f02324d8916a1ca5247a313da624e8365a57c438aefc58cc1301fd5ca5e0ff
3
+ metadata.gz: 98bd3e1d6ade2f96ff4ddbf4ae825e1926f56990b6552ff22aab6007546fd32e
4
+ data.tar.gz: edab64f8c8a7108a4614f4f5b7494ddb73611587c54bf6a4c270385303d9d53b
5
5
  SHA512:
6
- metadata.gz: fca3097df5295eaf1f3fa67efe04c7d1226c5c050d01f3e3c7319210967addfa01d98fd7a7b67024cae0a085f56b0357cf4b9b7ee4eb7aedd027b7996dc59356
7
- data.tar.gz: c45dc8053b73c26b7f0eaac877d35118aad007974ae61cb6b3e19aac373e8713f682513161e1af5250263bf461a03aa71846e4149fed411e81c771a2943c8e57
6
+ metadata.gz: 26c290fd4cd0d6d2a564b8e8dacaab4f584a0548c096999870b43dc755128faf23db58105ccbc16f424aac5384e79a51d07eada5e7a82e66b3d76abec69ad8e7
7
+ data.tar.gz: 1ee2fe285c913db6851ac97198d73c7159972c35c3c30e2b57342d226d0387e4940df19b7bd737769bfac3b2ab74fec3813d41a1f70c2d39e5202c20c35b8369
checksums.yaml.gz.sig CHANGED
Binary file
data/CHANGELOG.md CHANGED
@@ -1,5 +1,45 @@
1
1
  # Changes (Release notes)
2
2
 
3
+ * 4.14.0
4
+
5
+ * new: server: option `passphrase` for simpler command line (#114)
6
+ * new: percent selector for entities identifier
7
+ * new: faspex5: shared inbox and workgroup membership management
8
+ * new: faspex5: invite external user to shared inbox
9
+ * new: faspex5: package list and receive from workgroups and shared inboxes
10
+ * new: config: Command `ascp info` shows default transfer spec.
11
+ * new: httpgw: synchronous and asynchronous upload modes
12
+ * new: node: command `bandwidth_average` to get average bandwidth of node, per periods
13
+ * fix: option `ts`: deep add and remove of keys. (#117)
14
+ * fix: faspex5: user lookup for `packages send` shall be exact match (#120)
15
+ * fix: direct: if transfer spec contains "paths" and elements with "destination", but first element has only "source", then destinations were ignored. Now "destination" all or none is enforced.
16
+ * change: using `aoc files` or node gen4 operations (`browse`, `delete`) on a link will follow the link only if path ends with /
17
+ * change: shares: command `repository` is changed to `files` for consistency with aoc and upcoming faspex5, but is still available as alias
18
+ * change: aoc: better handling of shared links
19
+ * change: global: option `value` is deprecated. Use positional parameter for creation data and option `query` for list/delete operations.
20
+ * change: config: remove deprecated command: `export_to_cli`
21
+ * change: config: removed all legacy preset command, newer command `preset` shall be used now.
22
+ * change: config: SDK is now installed in $HOME/.aspera/sdk instead of $HOME/.aspera/ascli/sdk
23
+ * change(break): aoc/node: Simplification: gen4 operations: show modify permission thumbnail are now directly under node gen 4 command. Command `file` is suppressed. Option `path` is suppressed. The default expected argument is a path. To provide a file id, use selector syntax: %id:_file_id_
24
+ * change(break): node: option `token_type` is removed, as starting with HSTS 4.3 basic token is only allowed with access keys, so use gen4 operations: `acc do self`
25
+
26
+ * 4.13.0
27
+
28
+ * new: preview: option `reencode_ffmpeg` allows overriding all re-encoding options
29
+ * new: faspex5: package delete (#107)
30
+ * new: faspex5: package recv for inboxes and regular users (#108)
31
+ * new: faspex5: smtp management
32
+ * new: faspex5: use public link for authorization of package download, using option `link`
33
+ * new: faspex5: list content of package, and allow partial download of package
34
+ * new: faspex5: list packages support multiple pages and items limitations (`max` and `pmax`)
35
+ * new: aoc: files operations with workspace-less user (#109)
36
+ * new: node: async with gen3 token (#110)
37
+ * new: node: display of preview of file in terminal for access keys
38
+ * change: option `transfer_info` is now cumulative, setting several times merge values
39
+ * change(deprecation): Removed support of Ruby 2.4 and 2.5 : too old, no security update since a long time. If you need older ruby version use older gem version.
40
+ * fix: cos: do not use refresh token when not supported
41
+ * fix: container: SDK installed in other folder than `ascli` (#106)
42
+
3
43
  * 4.12.0
4
44
 
5
45
  * new: docker: build image from official gem version, possibility to deploy beta as well
@@ -172,7 +212,7 @@
172
212
  * new: agent `http_gw` now supports upload
173
213
  * new: added option `sdk_url` to install SDK from local file for offline install
174
214
  * new: check new gem version periodically
175
- * new: the --fields= option, support -_fieldname_ to remove a field from default fields
215
+ * new: the --fields= option, support -_field_name_ to remove a field from default fields
176
216
  * new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
177
217
  * new: mimemagic is now optional, needs manual install for `preview`, compatible with version 0.4.x
178
218
  * new: AoC a password can be provided for a public link
@@ -192,7 +232,7 @@
192
232
 
193
233
  * 0.11.8
194
234
 
195
- * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using Xvfb)
235
+ * Simplified to use `unoconv` instead of bare `libreoffice` for office conversion, as `unoconv` does not require a X server (previously using `Xvfb`)
196
236
 
197
237
  * 0.11.7
198
238
 
@@ -313,7 +353,7 @@
313
353
  * 0.10.4
314
354
 
315
355
  * new options for AoC : `secrets`
316
- * ACLI-533 temp file list folder to use file lists is set by default, and used by `asession`
356
+ * `ACLI-533` temp file list folder to use file lists is set by default, and used by `asession`
317
357
 
318
358
  * 0.10.3
319
359
 
@@ -353,13 +393,13 @@
353
393
 
354
394
  * new command to display basic token of node
355
395
  * new command to display bearer token of node in AoC
356
- * the --fields= option, support +_fieldname_ to add a field to default fields
396
+ * the --fields= option, support +_field_name_ to add a field to default fields
357
397
  * many small changes
358
398
 
359
399
  * 0.9.32
360
400
 
361
401
  * all Faspex public links are now supported
362
- * removed faspex operation recv_publink
402
+ * removed faspex operation `recv_publink`
363
403
  * replaced with option `link` (consistent with AoC)
364
404
 
365
405
  * 0.9.31
data/CONTRIBUTING.md CHANGED
@@ -1,5 +1,5 @@
1
1
  # Contributing
2
-
2
+ <!-- cspell:words passin GEMVERS -->
3
3
  ## Reporting Issues and Vulnerabilities
4
4
 
5
5
  You can report issues at <https://github.com/IBM/aspera-cli/issues>
@@ -34,6 +34,20 @@ If you want to contribute, please:
34
34
  - Send a pull request on GitHub.
35
35
  - run Rubocop to comply for coding standards
36
36
 
37
+ ## Architecture
38
+
39
+ A list of classes are provided in <docs/ml.png>
40
+
41
+ Architecture:
42
+
43
+ ![Architecture](docs/architecture.png)
44
+
45
+ The entry point is lib/aspera/cli/main.rb
46
+
47
+ Plugins are located in lib/aspera/cli/plugins
48
+
49
+ Transfer agents: lib/aspera/fasp
50
+
37
51
  ## Running Tests
38
52
 
39
53
  First, a testing environment must be created:
@@ -73,51 +87,111 @@ It is also possible to build a non-signed version for development purpose: `make
73
87
 
74
88
  ### Gem Signature
75
89
 
76
- Refer to: <https://guides.rubygems.org/security/>
90
+ Refer to:
77
91
 
78
- The gem is signed with the public certificate found in `certs` and a secret key (kept secret by maintainer).
92
+ - <https://guides.rubygems.org/security/>
93
+ - <https://ruby-doc.org/current/stdlibs/rubygems/Gem/Security.html>
94
+ - `gem cert --help`
79
95
 
80
- To build the signed gem:
96
+ Then procedure is as follows:
81
97
 
82
- ```bash
83
- SIGNING_KEY=/path/to/signing_key.pem make
84
- ```
98
+ - The maintainer creates the initial certificate and a private key:
85
99
 
86
- The user can activate gem signature verification on installation:
100
+ ```bash
101
+ cd /path/to/vault
102
+ gem cert --build maintainer@example.com
103
+ ```
87
104
 
88
- - Add the certificate to gem trusted certificates:
105
+ > **Note:** the email must match the field `spec.email` in `aspera-cli.gemspec`
89
106
 
90
- ```bash
91
- curl https://raw.githubusercontent.com/IBM/aspera-cli/main/certs/aspera-cli-public-cert.pem -so aspera-cli-certificate.pem
92
- gem cert --add aspera-cli-certificate.pem
93
- rm aspera-cli-certificate.pem
94
- ```
107
+ This creates two files in folder `/path/to/vault` (e.g. $HOME/.ssh):
108
+
109
+ - `gem-private_key.pem` : This file shall be kept secret in a vault.
110
+ - `gem-public_cert.pem` : This file is copied to a public place, here in folder `certs`
95
111
 
96
- - Install the gem with `HighSecurity` or `MediumSecurity`: this will succeed only of the gem is trusted.
112
+ > **Note:** Alternatively, use an existing key or generate one, and then `make new-cert`
97
113
 
98
- ```bash
99
- gem install -P HighSecurity aspera-cli
100
- ```
114
+ - The maintainer build the signed gem.
115
+
116
+ The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
117
+
118
+ To build the signed gem:
119
+
120
+ ```bash
121
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make
122
+ ```
123
+
124
+ - The user can activate gem signature verification on installation:
125
+
126
+ Add the certificate to gem trusted certificates:
127
+
128
+ ```bash
129
+ curl https://raw.githubusercontent.com/IBM/aspera-cli/main/certs/aspera-cli-public-cert.pem -so aspera-cli-certificate.pem
130
+ gem cert --add aspera-cli-certificate.pem
131
+ rm aspera-cli-certificate.pem
132
+ ```
133
+
134
+ - The user installs the gem with `HighSecurity` or `MediumSecurity`: this will succeed only of the gem is trusted.
135
+
136
+ ```bash
137
+ gem install -P HighSecurity aspera-cli
138
+ ```
139
+
140
+ - The maintainer can renew the certificate when it is expired using the same private key:
141
+
142
+ ```bash
143
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make update-cert
144
+ ```
145
+
146
+ Alternatively, to generate a new certificate with the same key:
147
+
148
+ ```bash
149
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make new-cert
150
+ ```
151
+
152
+ - Show the current certificate contents
153
+
154
+ ```bash
155
+ make show-cert
156
+ ```
157
+
158
+ > Note: to provide a passphrase add argument: `-passin pass:_value_` to `openssl`
159
+
160
+ - Check that the signing key is the same as used to sign the certificate:
161
+
162
+ ```bash
163
+ SIGNING_KEY=/path/to/vault/gem-private_key.pem make check-cert-key
164
+ ```
101
165
 
102
166
  ## Docker image build
103
167
 
104
- The default docker image build installs the official gem version directly from <rubygems.org>.
168
+ The Dockerfile template allows customizing the optional copy of gem file , versus install from rubygems,org as well as customizing the retrieval of the SDK.
169
+
170
+ ### Default image build
105
171
 
106
- By default it will build the image for the version in the current repository:
172
+ Build the image:
107
173
 
108
174
  ```bash
109
175
  make docker
110
176
  ```
111
177
 
178
+ This does the following:
179
+
180
+ - Install the official gem version directly from <rubygems.org>.
181
+ - Build the image for the version number in the current repository
182
+ - creates tags for both the version and `latest`
183
+
112
184
  > **Note:** This target creates the `Dockerfile` from an `ERB` (embedded Ruby) template (Makefile target `dockerfile`).
113
185
  A template is used as it allows some level of customization to tell where to take the gem from, as as for the SDK file.
114
186
 
115
- Then, to push to the image registry:
187
+ Then, to push to the image registry (both tags: version and `latest`):
116
188
 
117
189
  ```bash
118
190
  make dpush
119
191
  ```
120
192
 
193
+ ### Specific version image build
194
+
121
195
  It is possible to build a specific version by setting envvar `GEMVERS` and `make` with option `-e`:
122
196
 
123
197
  ```bash
@@ -125,14 +199,31 @@ GEMVERS=4.11.0 make -e docker
125
199
  GEMVERS=4.11.0 make -e dpush
126
200
  ```
127
201
 
202
+ ### Development version image build
203
+
128
204
  To build/push a beta/development container:
205
+ it does not create the `latest` tag, it uses the gem file generated locally.
129
206
 
130
207
  ```bash
131
208
  make dockerbeta
132
209
  make dpushversion
133
210
  ```
134
211
 
135
- The Dockerfile template allows customizing the optional copy of gem file , versus install from rubygems,org as well as customizing the retrieval of the SDK.
212
+ ## Single executable build
213
+
214
+ It is possible to build `ascli` as a single native executable using `rubyc` (gem `ruby-packer`).
215
+
216
+ ```bash
217
+ make single
218
+ ```
219
+
220
+ This is based on tool `rubyc` (gem `ruby-packer`): either place the executable in folder `tmp`, or the architecture-specific version will be automatically downloaded.
221
+
222
+ > **Note:** Perl Findbin is needed, install on Linux with:
223
+
224
+ ```bash
225
+ sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
226
+ ```
136
227
 
137
228
  ## Long Term Implementation and delivery improvements
138
229