aspera-cli 4.24.2 → 4.25.0.pre2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (73) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/CHANGELOG.md +1067 -758
  4. data/CONTRIBUTING.md +93 -120
  5. data/README.md +817 -510
  6. data/lib/aspera/agent/direct.rb +14 -12
  7. data/lib/aspera/agent/transferd.rb +4 -4
  8. data/lib/aspera/api/aoc.rb +71 -43
  9. data/lib/aspera/api/cos_node.rb +3 -2
  10. data/lib/aspera/api/faspex.rb +6 -5
  11. data/lib/aspera/api/node.rb +10 -12
  12. data/lib/aspera/ascmd.rb +1 -2
  13. data/lib/aspera/ascp/installation.rb +55 -41
  14. data/lib/aspera/ascp/management.rb +9 -5
  15. data/lib/aspera/assert.rb +28 -6
  16. data/lib/aspera/cli/error.rb +4 -2
  17. data/lib/aspera/cli/extended_value.rb +94 -62
  18. data/lib/aspera/cli/formatter.rb +55 -22
  19. data/lib/aspera/cli/main.rb +21 -14
  20. data/lib/aspera/cli/manager.rb +349 -248
  21. data/lib/aspera/cli/plugins/alee.rb +3 -3
  22. data/lib/aspera/cli/plugins/aoc.rb +94 -51
  23. data/lib/aspera/cli/plugins/base.rb +62 -49
  24. data/lib/aspera/cli/plugins/config.rb +85 -96
  25. data/lib/aspera/cli/plugins/console.rb +15 -9
  26. data/lib/aspera/cli/plugins/cos.rb +1 -1
  27. data/lib/aspera/cli/plugins/faspex.rb +34 -27
  28. data/lib/aspera/cli/plugins/faspex5.rb +47 -44
  29. data/lib/aspera/cli/plugins/faspio.rb +7 -6
  30. data/lib/aspera/cli/plugins/httpgw.rb +3 -2
  31. data/lib/aspera/cli/plugins/node.rb +132 -120
  32. data/lib/aspera/cli/plugins/oauth.rb +1 -1
  33. data/lib/aspera/cli/plugins/orchestrator.rb +116 -33
  34. data/lib/aspera/cli/plugins/preview.rb +26 -46
  35. data/lib/aspera/cli/plugins/server.rb +9 -10
  36. data/lib/aspera/cli/plugins/shares.rb +77 -43
  37. data/lib/aspera/cli/sync_actions.rb +49 -38
  38. data/lib/aspera/cli/transfer_agent.rb +16 -34
  39. data/lib/aspera/cli/version.rb +1 -1
  40. data/lib/aspera/cli/wizard.rb +8 -5
  41. data/lib/aspera/command_line_builder.rb +20 -17
  42. data/lib/aspera/coverage.rb +6 -2
  43. data/lib/aspera/environment.rb +71 -84
  44. data/lib/aspera/faspex_gw.rb +1 -1
  45. data/lib/aspera/faspex_postproc.rb +1 -1
  46. data/lib/aspera/keychain/factory.rb +1 -2
  47. data/lib/aspera/keychain/macos_security.rb +2 -2
  48. data/lib/aspera/log.rb +2 -1
  49. data/lib/aspera/markdown.rb +31 -0
  50. data/lib/aspera/nagios.rb +6 -5
  51. data/lib/aspera/oauth/base.rb +17 -27
  52. data/lib/aspera/oauth/factory.rb +1 -1
  53. data/lib/aspera/oauth/url_json.rb +2 -1
  54. data/lib/aspera/preview/file_types.rb +23 -37
  55. data/lib/aspera/preview/terminal.rb +95 -29
  56. data/lib/aspera/preview/utils.rb +6 -5
  57. data/lib/aspera/products/connect.rb +3 -3
  58. data/lib/aspera/rest.rb +51 -39
  59. data/lib/aspera/rest_error_analyzer.rb +4 -4
  60. data/lib/aspera/ssh.rb +5 -2
  61. data/lib/aspera/ssl.rb +41 -0
  62. data/lib/aspera/sync/conf.schema.yaml +182 -34
  63. data/lib/aspera/sync/database.rb +2 -1
  64. data/lib/aspera/sync/operations.rb +128 -72
  65. data/lib/aspera/transfer/parameters.rb +3 -4
  66. data/lib/aspera/transfer/spec.rb +2 -3
  67. data/lib/aspera/transfer/spec.schema.yaml +49 -19
  68. data/lib/aspera/transfer/spec_doc.rb +14 -14
  69. data/lib/aspera/uri_reader.rb +1 -1
  70. data/lib/transferd_pb.rb +2 -2
  71. data.tar.gz.sig +0 -0
  72. metadata +33 -6
  73. metadata.gz.sig +0 -0
data/CONTRIBUTING.md CHANGED
@@ -1,8 +1,8 @@
1
1
  # Contributing
2
- <!-- cspell:words passin -->
2
+
3
3
  ## Reporting Issues and Vulnerabilities
4
4
 
5
- If you encounter a problem or vulnerability, please report it at [GitHub Issues](https://github.com/IBM/aspera-cli/issues).
5
+ If you encounter a problem or a security vulnerability, please report it on [GitHub Issues](https://github.com/IBM/aspera-cli/issues).
6
6
 
7
7
  Before submitting a new issue:
8
8
 
@@ -10,30 +10,32 @@ Before submitting a new issue:
10
10
 
11
11
  To help us assist you efficiently, include the following in your report:
12
12
 
13
- - The version of ascli you are using:
13
+ - The version of `ascli` you are using:
14
14
 
15
15
  ```bash
16
16
  ascli version
17
17
  ```
18
18
 
19
- - Confirmation that you're using the latest version (update if not).
20
- - The output of your Ruby environment:
19
+ - Confirmation that you are using the latest version (update it if needed).
20
+ - Your Ruby version information:
21
21
 
22
22
  ```bash
23
- ruby -e "puts RUBY_DESCRIPTION"
23
+ ruby -v
24
24
  ```
25
25
 
26
26
  ## Making Contributions
27
27
 
28
- We welcome contributions to improve the aspera-cli project!
28
+ We welcome contributions to improve the `aspera-cli` project!
29
29
 
30
30
  ### Getting Started
31
31
 
32
- Clone the repository and navigate to the project folder:
32
+ Clone the repository and navigate to the project's root directory:
33
33
 
34
34
  ```bash
35
35
  git clone https://github.com/IBM/aspera-cli.git
36
36
  cd aspera-cli
37
+ bundle install
38
+ bundle exec rake -T
37
39
  ```
38
40
 
39
41
  For testing instructions, refer to [Running Tests](#running-tests).
@@ -48,7 +50,7 @@ To submit a contribution:
48
50
 
49
51
  1. **Implement** your feature or bug fix.
50
52
 
51
- 1. **Write tests** to ensure your changes are robust and won’t break future versions.
53
+ 1. **Write tests** to ensure your changes are robust and prevent regressions.
52
54
 
53
55
  1. Run `rubocop` to ensure your code follows the Ruby style guide.
54
56
 
@@ -61,68 +63,64 @@ To submit a contribution:
61
63
 
62
64
  ## Architecture
63
65
 
64
- The overall architecture of aspera-cli is modular and extensible.
66
+ The overall architecture of `aspera-cli` is modular and extensible.
65
67
 
66
68
  ![Architecture](docs/architecture.png)
67
69
 
68
- Structure Highlights:
70
+ ### Structure Highlights
69
71
 
70
72
  - Entry Point:
71
73
 
72
- `lib/aspera/cli/main.rb` - This is where the CLI execution begins.
74
+ `lib/aspera/cli/main.rb`, CLI startup logic.
73
75
 
74
76
  - Plugins:
75
77
 
76
- Located in `lib/aspera/cli/plugins`, plugins extend CLI functionality and encapsulate specific features.
78
+ Located in `lib/aspera/cli/plugins`; plugins extend CLI functionality and encapsulate specific features.
77
79
 
78
80
  - Transfer Agents:
79
81
 
80
- Found in `lib/aspera/agent`, these handle data transfers operations.
82
+ Found in `lib/aspera/agent`, these handle data transfer operations.
81
83
 
82
- A list of classes are provided in <docs/uml.png>
84
+ Class diagrams are provided in <docs/uml.png>
83
85
 
84
- ## Ruby version
86
+ ## Ruby Environment
85
87
 
86
- `aspera-cli` is built with Ruby.
87
- You can install Ruby using any method you prefer (e.g., rbenv, rvm, system package manager).
88
+ `aspera-cli` is written in Ruby.
89
+ You can install Ruby using any method you prefer (e.g., `rbenv`, `rvm`, system package manager).
88
90
 
89
- To start with a clean slate and remove all installed gems:
91
+ To start with a clean state and remove all installed gems:
90
92
 
91
93
  ```bash
92
- make clean_gems
94
+ bundle exec rake tools:clean_gems
93
95
  ```
94
96
 
95
97
  > [!TIP]
96
98
  > This is especially useful before testing across different Ruby versions or preparing for a release.
97
99
 
98
- ## Tool chain
99
-
100
- TODO: document installation of tool chain.
100
+ ## Toolchain
101
101
 
102
- Build system uses GNU Make.
102
+ The build system uses Ruby's `rake`.
103
103
 
104
104
  ### Environment
105
105
 
106
- A few macros/env vars control some aspects:
106
+ A few macros and environment variables control certain aspects of the build:
107
107
 
108
- | `make` macro or Env var | Description |
109
- |-----------------------------|--------------------------------------|
110
- | `ASPERA_CLI_TEST_CONF_FILE` | Path to configuration file with secrets for tests |
111
- | `ASPERA_CLI_TEST_MACOS` | Set to `true` if local HSTS running on macOS |
112
- | `ASPERA_CLI_TEST_PRIVATE` | Path to private folder |
113
- | `ASPERA_CLI_DOC_CHECK_LINKS`| Check links still exist during doc generation. |
114
- | `ASPERA_CLI_DOC_DEBUG` | Enable debug in doc generation. |
115
- | `ENABLE_COVERAGE` | Tests with coverage analysis if set. |
116
- | `SIGNING_KEY` | Path to signing key to build Gem. |
117
- | `GEM_VERSION` | Gem version to build container |
108
+ | Environment variable | Description |
109
+ |-----------------------------|-----------------------------------------------------|
110
+ | `ASPERA_CLI_TEST_CONF_FILE` | Path to configuration file with secrets for tests. |
111
+ | `ASPERA_CLI_DOC_CHECK_LINKS`| Check links still exist during doc generation. |
112
+ | `LOG_LEVEL` | Change log level in `rake` tasks. |
113
+ | `ENABLE_COVERAGE` | Enable test coverage analysis when set. |
114
+ | `SIGNING_KEY` | Path to the signing key used to build the gem file. |
115
+ | `GEM_VERSION` | Override gem version for builds. |
118
116
 
119
- Those macros can be set either in an env var, or on the `make` command line.
117
+ These can be set either as environment variables or directly on the `rake` command line.
120
118
 
121
119
  > [!NOTE]
122
- > Env vars `ASPERA_CLI_TEST_*` are typically set in user's shell profile for development.
123
- > Others are more for "one shot" use.
120
+ > Environment variables `ASPERA_CLI_*` are typically set in the users shell profile for development.
121
+ > Others are intended for one‑time use on the command line.
124
122
 
125
- To use the CLI directly from the development environment, add this to your shell profile:
123
+ To use the CLI directly from the development environment, add this to your shell profile (adapt the real path):
126
124
 
127
125
  ```bash
128
126
  dev_ascli=$HOME/github/aspera-cli
@@ -134,9 +132,9 @@ export RUBYLIB=$dev_ascli/lib:$RUBYLIB
134
132
 
135
133
  Documentation is generated with `pandoc` and `LaTeX`.
136
134
 
137
- IBM font `Plex` is used, for installation see [IBM Plex](https://www.ibm.com/plex/).
135
+ The IBM `Plex` font is used; for installation instructions, see [IBM Plex](https://www.ibm.com/plex/).
138
136
 
139
- On macOS to install `lualatex` and all packages:
137
+ On macOS, to install `lualatex` and all packages:
140
138
 
141
139
  ```bash
142
140
  brew install texlive
@@ -149,138 +147,113 @@ tlmgr update --self
149
147
  tlmgr install fvextra selnolig lualatex-math
150
148
  ```
151
149
 
152
- To check URL during doc generation, set env var: `ASPERA_CLI_DOC_CHECK_LINKS=1`.
150
+ To check URLs during documentation generation, set the environment variable: `ASPERA_CLI_DOC_CHECK_LINKS=1`.
151
+
152
+ To debug documentation generation, set the environment variable: `ASPERA_CLI_DOC_DEBUG=debug`.
153
+
154
+ ## Test Environment
153
155
 
154
- To debug doc generation, set env var: `ASPERA_CLI_DOC_DEBUG=debug`.
156
+ Refer to <tests/README.md>.
155
157
 
156
- ## Running Tests
158
+ ## Build
157
159
 
158
- First, a testing configuration file must be created.
159
- From project top folder, execute:
160
+ The gem is built with:
160
161
 
161
162
  ```bash
162
- mkdir ~/some_secure_folder
163
- cp docs/test_env.conf ~/some_secure_folder/.
163
+ bundle install
164
+ bundle exec rake build
164
165
  ```
165
166
 
166
- Fill `~/some_secure_folder/test_env.conf` with system URLs and credentials for tests.
167
-
168
- Then, tell where this file is located:
167
+ If you don't want to install optional gems:
169
168
 
170
169
  ```bash
171
- export ASPERA_CLI_TEST_CONF_FILE=~/some_secure_folder/test_env.conf
170
+ bundle config set without optional
172
171
  ```
173
172
 
174
- This project uses a `Makefile` for tests, in main folder:
173
+ To build a beta version:
175
174
 
176
175
  ```bash
177
- make test
176
+ export GEM_VERSION=$(env -u GEM_VERSION rake tools:version).$(date +%Y%m%d%H%M)
178
177
  ```
179
178
 
180
- When new commands are added to the CLI, new tests shall be added to the test suite in `tests/Makefile`.
179
+ ### Signed gem
181
180
 
182
- One can also go to the `tests` folder:
181
+ A private key is required to generate a signed gem.
182
+ Its path must be set using environment variable `SIGNING_KEY`.
183
+ The gem is signed with the public certificate found in `certs` and the private key specified by `SIGNING_KEY` (kept secret by the maintainer).
183
184
 
184
185
  ```bash
185
- cd tests
186
- make
187
- make full
188
- make list
189
- make torc
190
- make skip_torc
191
- make skip T=orch_wizard
186
+ bundle exec rake build SIGNING_KEY=/path/to/vault/gem-private_key.pem
192
187
  ```
193
188
 
194
- ### Special tests
189
+ Refer to <certs/README.md>.
195
190
 
196
- Some gems are optional: `rmagick` and `grpc`, as they require compilation of native code which may cause problems.
197
- By default, tests that use those gems are skipped.
198
- To run them: `make optional`.
199
- Those tests also require the optional gems to be installed: `make install_optional_gems`.
191
+ ### gRPC stubs for transfer SDK
200
192
 
201
- Some other tests require interactive input. To run them: `make interactive`
193
+ Update with:
202
194
 
203
- To run every test: `make full`
195
+ ```bash
196
+ bundle exec rake tools:grpc
197
+ ```
204
198
 
205
- ### Pre-release tests
199
+ It downloads the latest `proto` file and then compiles it.
206
200
 
207
- For preparation of a release, do the following:
201
+ ## Container image build
208
202
 
209
- 1. Select a ruby version to test with.
210
- 2. Remove all gems: `make clean_gems`
211
- 3. `cd tests && make full`
203
+ See [Container build](./container/README.md).
212
204
 
213
- To test additional Ruby version, repeat the procedure with other Ruby versions.
205
+ ## Single executable build
214
206
 
215
- ## Coverage
207
+ See [Executable build](build/binary/README.md).
216
208
 
217
- A coverage report can be generated in folder `coverage` using gem `SimpleCov`.
218
- Enable coverage monitoring using macro/envvar `ENABLE_COVERAGE`.
209
+ To list operations:
219
210
 
220
211
  ```bash
221
- cd tests
222
- make ENABLE_COVERAGE=1
212
+ bundle exec rake -T ^binary:
223
213
  ```
224
214
 
225
- Once tests are completed, or during test, consult the page: [coverage/index.html](coverage/index.html)
215
+ ## Release
226
216
 
227
- ## Build
217
+ ### Checklist before a new release
218
+
219
+ When preparing for a new release, do the following:
228
220
 
229
- By default, the gem is built signed: `make`.
230
- A private key is required to generate a signed Gem.
231
- Its path must be set using macro/envvar `SIGNING_KEY`, see below.
232
- The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
221
+ - Run the test suite:
233
222
 
234
223
  ```bash
235
- make SIGNING_KEY=/path/to/vault/gem-private_key.pem
224
+ bundle exec rake test:run
236
225
  ```
237
226
 
238
- It is also possible to build an unsigned version for development purpose: `make unsigned_gem`.
239
-
240
- ### Gem Signature
241
-
242
- Refer to <certs/README.md>
243
-
244
- ### gRPC stubs for transfer SDK
245
-
246
- Update with:
227
+ - Set beta version (to use the gem file built):
247
228
 
248
229
  ```bash
249
- make grpc
230
+ export GEM_VERSION=$(env -u GEM_VERSION rake tools:version).$(date +%Y%m%d%H%M)
250
231
  ```
251
232
 
252
- It downloads the latest proto file and then compiles it.
253
-
254
- ## Container image build
255
-
256
- See [Container build](./container/README.md).
257
-
258
- For operations, move to the folder:
233
+ - Verify that the container builds successfully (using the beta version):
259
234
 
260
235
  ```bash
261
- cd container
236
+ bundle exec rake container:build
237
+ bundle exec rake container:test
262
238
  ```
263
239
 
264
- ## Single executable build
240
+ ### Version Release Process
265
241
 
266
- See [Executable build](./binary/README.md).
242
+ Once the development branch is ready for release:
267
243
 
268
- For operations, move to the folder:
269
-
270
- ```bash
271
- cd binary
272
- ```
244
+ - Decide on the version number
273
245
 
274
- ## Development check list for new release
246
+ - Update the version in `lib/aspera/cli/version.rb`
275
247
 
276
- When preparing for a new release do the following:
248
+ - Execute `bundle exec rake release` (see `.github/workflows/deploy.yml`)
277
249
 
278
- - TODO
250
+ - It builds the `.gem` file in the `pkg` folder.
251
+ - Then runs `gem push`.
279
252
 
280
- ## Long Term Implementation and delivery improvements
253
+ ## LongTerm Implementation and Delivery Improvements
281
254
 
282
- - replace Rest and OAuth classes with ruby standard gems:
255
+ - Replace custom REST and OAuth classes with standard Ruby gems:
283
256
  - <https://github.com/rest-client/rest-client>
284
257
  - <https://github.com/oauth-xx/oauth2>
285
- - use gem Thor <http://whatisthor.com/> (or other standard Ruby CLI manager)
286
- - look at <https://github.com/phusion/traveling-ruby>
258
+ - Use the `thor` gem <http://whatisthor.com/> (or other standard Ruby CLI manager)
259
+ - Look at <https://github.com/phusion/traveling-ruby>