aspera-cli 4.24.2 → 4.25.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (81) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/CHANGELOG.md +1070 -758
  4. data/CONTRIBUTING.md +130 -115
  5. data/README.md +961 -623
  6. data/lib/aspera/agent/direct.rb +14 -12
  7. data/lib/aspera/agent/factory.rb +9 -6
  8. data/lib/aspera/agent/transferd.rb +8 -8
  9. data/lib/aspera/api/aoc.rb +104 -67
  10. data/lib/aspera/api/ats.rb +1 -0
  11. data/lib/aspera/api/cos_node.rb +3 -2
  12. data/lib/aspera/api/faspex.rb +17 -10
  13. data/lib/aspera/api/node.rb +10 -12
  14. data/lib/aspera/ascmd.rb +2 -3
  15. data/lib/aspera/ascp/installation.rb +60 -46
  16. data/lib/aspera/ascp/management.rb +9 -5
  17. data/lib/aspera/assert.rb +28 -6
  18. data/lib/aspera/cli/error.rb +4 -2
  19. data/lib/aspera/cli/extended_value.rb +94 -62
  20. data/lib/aspera/cli/formatter.rb +44 -58
  21. data/lib/aspera/cli/main.rb +21 -14
  22. data/lib/aspera/cli/manager.rb +317 -250
  23. data/lib/aspera/cli/plugins/alee.rb +3 -3
  24. data/lib/aspera/cli/plugins/aoc.rb +139 -78
  25. data/lib/aspera/cli/plugins/ats.rb +30 -36
  26. data/lib/aspera/cli/plugins/base.rb +68 -55
  27. data/lib/aspera/cli/plugins/config.rb +90 -100
  28. data/lib/aspera/cli/plugins/console.rb +15 -9
  29. data/lib/aspera/cli/plugins/cos.rb +1 -1
  30. data/lib/aspera/cli/plugins/faspex.rb +39 -30
  31. data/lib/aspera/cli/plugins/faspex5.rb +57 -52
  32. data/lib/aspera/cli/plugins/faspio.rb +10 -7
  33. data/lib/aspera/cli/plugins/httpgw.rb +3 -2
  34. data/lib/aspera/cli/plugins/node.rb +140 -125
  35. data/lib/aspera/cli/plugins/oauth.rb +13 -12
  36. data/lib/aspera/cli/plugins/orchestrator.rb +116 -33
  37. data/lib/aspera/cli/plugins/preview.rb +28 -48
  38. data/lib/aspera/cli/plugins/server.rb +9 -10
  39. data/lib/aspera/cli/plugins/shares.rb +77 -43
  40. data/lib/aspera/cli/sync_actions.rb +49 -38
  41. data/lib/aspera/cli/transfer_agent.rb +16 -35
  42. data/lib/aspera/cli/version.rb +1 -1
  43. data/lib/aspera/cli/wizard.rb +8 -5
  44. data/lib/aspera/command_line_builder.rb +24 -21
  45. data/lib/aspera/coverage.rb +6 -2
  46. data/lib/aspera/dot_container.rb +108 -0
  47. data/lib/aspera/environment.rb +71 -84
  48. data/lib/aspera/faspex_gw.rb +1 -1
  49. data/lib/aspera/faspex_postproc.rb +1 -1
  50. data/lib/aspera/id_generator.rb +7 -10
  51. data/lib/aspera/keychain/factory.rb +1 -2
  52. data/lib/aspera/keychain/macos_security.rb +2 -2
  53. data/lib/aspera/log.rb +2 -1
  54. data/lib/aspera/markdown.rb +31 -0
  55. data/lib/aspera/nagios.rb +6 -5
  56. data/lib/aspera/oauth/base.rb +41 -64
  57. data/lib/aspera/oauth/factory.rb +6 -7
  58. data/lib/aspera/oauth/generic.rb +1 -1
  59. data/lib/aspera/oauth/jwt.rb +1 -1
  60. data/lib/aspera/oauth/url_json.rb +6 -4
  61. data/lib/aspera/oauth/web.rb +2 -2
  62. data/lib/aspera/preview/file_types.rb +24 -38
  63. data/lib/aspera/preview/terminal.rb +95 -29
  64. data/lib/aspera/preview/utils.rb +6 -5
  65. data/lib/aspera/products/connect.rb +3 -3
  66. data/lib/aspera/rest.rb +54 -39
  67. data/lib/aspera/rest_error_analyzer.rb +4 -4
  68. data/lib/aspera/ssh.rb +10 -6
  69. data/lib/aspera/ssl.rb +41 -0
  70. data/lib/aspera/sync/conf.schema.yaml +184 -36
  71. data/lib/aspera/sync/database.rb +2 -1
  72. data/lib/aspera/sync/operations.rb +128 -72
  73. data/lib/aspera/transfer/parameters.rb +9 -10
  74. data/lib/aspera/transfer/spec.rb +2 -3
  75. data/lib/aspera/transfer/spec.schema.yaml +52 -22
  76. data/lib/aspera/transfer/spec_doc.rb +20 -30
  77. data/lib/aspera/uri_reader.rb +18 -4
  78. data/lib/transferd_pb.rb +2 -2
  79. data.tar.gz.sig +0 -0
  80. metadata +34 -6
  81. metadata.gz.sig +0 -0
data/CONTRIBUTING.md CHANGED
@@ -1,8 +1,8 @@
1
1
  # Contributing
2
- <!-- cspell:words passin -->
2
+
3
3
  ## Reporting Issues and Vulnerabilities
4
4
 
5
- If you encounter a problem or vulnerability, please report it at [GitHub Issues](https://github.com/IBM/aspera-cli/issues).
5
+ If you encounter a problem or a security vulnerability, please report it on [GitHub Issues](https://github.com/IBM/aspera-cli/issues).
6
6
 
7
7
  Before submitting a new issue:
8
8
 
@@ -10,30 +10,32 @@ Before submitting a new issue:
10
10
 
11
11
  To help us assist you efficiently, include the following in your report:
12
12
 
13
- - The version of ascli you are using:
13
+ - The version of `ascli` you are using:
14
14
 
15
15
  ```bash
16
16
  ascli version
17
17
  ```
18
18
 
19
- - Confirmation that you're using the latest version (update if not).
20
- - The output of your Ruby environment:
19
+ - Confirmation that you are using the latest version (update it if needed).
20
+ - Your Ruby version information:
21
21
 
22
22
  ```bash
23
- ruby -e "puts RUBY_DESCRIPTION"
23
+ ruby -v
24
24
  ```
25
25
 
26
26
  ## Making Contributions
27
27
 
28
- We welcome contributions to improve the aspera-cli project!
28
+ We welcome contributions to improve the `aspera-cli` project!
29
29
 
30
30
  ### Getting Started
31
31
 
32
- Clone the repository and navigate to the project folder:
32
+ Clone the repository and navigate to the project's root directory:
33
33
 
34
34
  ```bash
35
35
  git clone https://github.com/IBM/aspera-cli.git
36
36
  cd aspera-cli
37
+ bundle install
38
+ bundle exec rake -T
37
39
  ```
38
40
 
39
41
  For testing instructions, refer to [Running Tests](#running-tests).
@@ -48,7 +50,7 @@ To submit a contribution:
48
50
 
49
51
  1. **Implement** your feature or bug fix.
50
52
 
51
- 1. **Write tests** to ensure your changes are robust and won’t break future versions.
53
+ 1. **Write tests** to ensure your changes are robust and prevent regressions.
52
54
 
53
55
  1. Run `rubocop` to ensure your code follows the Ruby style guide.
54
56
 
@@ -61,68 +63,64 @@ To submit a contribution:
61
63
 
62
64
  ## Architecture
63
65
 
64
- The overall architecture of aspera-cli is modular and extensible.
66
+ The overall architecture of `aspera-cli` is modular and extensible.
65
67
 
66
68
  ![Architecture](docs/architecture.png)
67
69
 
68
- Structure Highlights:
70
+ ### Structure Highlights
69
71
 
70
72
  - Entry Point:
71
73
 
72
- `lib/aspera/cli/main.rb` - This is where the CLI execution begins.
74
+ `lib/aspera/cli/main.rb`, CLI startup logic.
73
75
 
74
76
  - Plugins:
75
77
 
76
- Located in `lib/aspera/cli/plugins`, plugins extend CLI functionality and encapsulate specific features.
78
+ Located in `lib/aspera/cli/plugins`; plugins extend CLI functionality and encapsulate specific features.
77
79
 
78
80
  - Transfer Agents:
79
81
 
80
- Found in `lib/aspera/agent`, these handle data transfers operations.
82
+ Found in `lib/aspera/agent`, these handle data transfer operations.
81
83
 
82
- A list of classes are provided in <docs/uml.png>
84
+ Class diagrams are provided in <docs/uml.png>
83
85
 
84
- ## Ruby version
86
+ ## Ruby Environment
85
87
 
86
- `aspera-cli` is built with Ruby.
87
- You can install Ruby using any method you prefer (e.g., rbenv, rvm, system package manager).
88
+ `aspera-cli` is written in Ruby.
89
+ You can install Ruby using any method you prefer (e.g., `rbenv`, `rvm`, system package manager).
88
90
 
89
- To start with a clean slate and remove all installed gems:
91
+ To start with a clean state and remove all installed gems:
90
92
 
91
93
  ```bash
92
- make clean_gems
94
+ bundle exec rake tools:clean_gems
93
95
  ```
94
96
 
95
97
  > [!TIP]
96
98
  > This is especially useful before testing across different Ruby versions or preparing for a release.
97
99
 
98
- ## Tool chain
99
-
100
- TODO: document installation of tool chain.
100
+ ## Toolchain
101
101
 
102
- Build system uses GNU Make.
102
+ The build system uses Ruby's `rake`.
103
103
 
104
104
  ### Environment
105
105
 
106
- A few macros/env vars control some aspects:
106
+ A few macros and environment variables control certain aspects of the build:
107
107
 
108
- | `make` macro or Env var | Description |
109
- |-----------------------------|--------------------------------------|
110
- | `ASPERA_CLI_TEST_CONF_FILE` | Path to configuration file with secrets for tests |
111
- | `ASPERA_CLI_TEST_MACOS` | Set to `true` if local HSTS running on macOS |
112
- | `ASPERA_CLI_TEST_PRIVATE` | Path to private folder |
113
- | `ASPERA_CLI_DOC_CHECK_LINKS`| Check links still exist during doc generation. |
114
- | `ASPERA_CLI_DOC_DEBUG` | Enable debug in doc generation. |
115
- | `ENABLE_COVERAGE` | Tests with coverage analysis if set. |
116
- | `SIGNING_KEY` | Path to signing key to build Gem. |
117
- | `GEM_VERSION` | Gem version to build container |
108
+ | Environment variable | Description |
109
+ |-----------------------------|-----------------------------------------------------|
110
+ | `ASPERA_CLI_TEST_CONF_URL` | URL for configuration file with secrets for tests. |
111
+ | `ASPERA_CLI_DOC_CHECK_LINKS`| Check links still exist during doc generation. |
112
+ | `LOG_LEVEL` | Change log level in `rake` tasks. |
113
+ | `ENABLE_COVERAGE` | Enable test coverage analysis when set. |
114
+ | `SIGNING_KEY` | Path to the signing key used to build the gem file. |
115
+ | `GEM_VERSION` | Override gem version for builds. |
118
116
 
119
- Those macros can be set either in an env var, or on the `make` command line.
117
+ These can be set either as environment variables or directly on the `rake` command line.
120
118
 
121
119
  > [!NOTE]
122
- > Env vars `ASPERA_CLI_TEST_*` are typically set in user's shell profile for development.
123
- > Others are more for "one shot" use.
120
+ > Environment variables `ASPERA_CLI_*` are typically set in the users shell profile for development.
121
+ > Others are intended for use on the command line.
124
122
 
125
- To use the CLI directly from the development environment, add this to your shell profile:
123
+ To use the CLI directly from the development environment, add this to your shell profile (adapt the real path):
126
124
 
127
125
  ```bash
128
126
  dev_ascli=$HOME/github/aspera-cli
@@ -134,9 +132,9 @@ export RUBYLIB=$dev_ascli/lib:$RUBYLIB
134
132
 
135
133
  Documentation is generated with `pandoc` and `LaTeX`.
136
134
 
137
- IBM font `Plex` is used, for installation see [IBM Plex](https://www.ibm.com/plex/).
135
+ The IBM `Plex` font is used; for installation instructions, see [IBM Plex](https://www.ibm.com/plex/).
138
136
 
139
- On macOS to install `lualatex` and all packages:
137
+ On macOS, to install `lualatex` and all packages:
140
138
 
141
139
  ```bash
142
140
  brew install texlive
@@ -149,138 +147,155 @@ tlmgr update --self
149
147
  tlmgr install fvextra selnolig lualatex-math
150
148
  ```
151
149
 
152
- To check URL during doc generation, set env var: `ASPERA_CLI_DOC_CHECK_LINKS=1`.
150
+ To check URLs during documentation generation, set the environment variable: `ASPERA_CLI_DOC_CHECK_LINKS=1`.
151
+
152
+ To debug documentation generation, set the environment variable: `ASPERA_CLI_DOC_DEBUG=debug`.
153
153
 
154
- To debug doc generation, set env var: `ASPERA_CLI_DOC_DEBUG=debug`.
154
+ ## Test Environment
155
155
 
156
- ## Running Tests
156
+ Refer to <tests/README.md>.
157
+
158
+ ## Build
157
159
 
158
- First, a testing configuration file must be created.
159
- From project top folder, execute:
160
+ The unsigned gem is built with:
160
161
 
161
162
  ```bash
162
- mkdir ~/some_secure_folder
163
- cp docs/test_env.conf ~/some_secure_folder/.
163
+ bundle install
164
+ bundle exec rake unsigned
164
165
  ```
165
166
 
166
- Fill `~/some_secure_folder/test_env.conf` with system URLs and credentials for tests.
167
-
168
- Then, tell where this file is located:
167
+ If you don't want to install optional gems:
169
168
 
170
169
  ```bash
171
- export ASPERA_CLI_TEST_CONF_FILE=~/some_secure_folder/test_env.conf
170
+ bundle config set without optional
172
171
  ```
173
172
 
174
- This project uses a `Makefile` for tests, in main folder:
173
+ To build a beta version:
175
174
 
176
175
  ```bash
177
- make test
176
+ export GEM_VERSION=$(env -u GEM_VERSION rake tools:version).$(date +%Y%m%d%H%M)
178
177
  ```
179
178
 
180
- When new commands are added to the CLI, new tests shall be added to the test suite in `tests/Makefile`.
179
+ ### Signed gem
181
180
 
182
- One can also go to the `tests` folder:
181
+ A private key is required to generate a signed gem.
182
+ Its path must be set using environment variable `SIGNING_KEY`.
183
+ The gem is signed with the public certificate found in `certs` and the private key specified by `SIGNING_KEY` (kept secret by the maintainer).
183
184
 
184
185
  ```bash
185
- cd tests
186
- make
187
- make full
188
- make list
189
- make torc
190
- make skip_torc
191
- make skip T=orch_wizard
186
+ bundle exec rake SIGNING_KEY=/path/to/vault/gem-private_key.pem
192
187
  ```
193
188
 
194
- ### Special tests
189
+ Refer to <certs/README.md>.
195
190
 
196
- Some gems are optional: `rmagick` and `grpc`, as they require compilation of native code which may cause problems.
197
- By default, tests that use those gems are skipped.
198
- To run them: `make optional`.
199
- Those tests also require the optional gems to be installed: `make install_optional_gems`.
191
+ ### gRPC stubs for transfer SDK
200
192
 
201
- Some other tests require interactive input. To run them: `make interactive`
193
+ Update with:
202
194
 
203
- To run every test: `make full`
195
+ ```bash
196
+ bundle exec rake tools:grpc
197
+ ```
204
198
 
205
- ### Pre-release tests
199
+ It downloads the latest `proto` file and then compiles it.
206
200
 
207
- For preparation of a release, do the following:
201
+ ## Container image build
208
202
 
209
- 1. Select a ruby version to test with.
210
- 2. Remove all gems: `make clean_gems`
211
- 3. `cd tests && make full`
203
+ See [Container build](./container/README.md).
212
204
 
213
- To test additional Ruby version, repeat the procedure with other Ruby versions.
205
+ ## Single executable build
214
206
 
215
- ## Coverage
207
+ See [Executable build](build/binary/README.md).
216
208
 
217
- A coverage report can be generated in folder `coverage` using gem `SimpleCov`.
218
- Enable coverage monitoring using macro/envvar `ENABLE_COVERAGE`.
209
+ To list operations:
219
210
 
220
211
  ```bash
221
- cd tests
222
- make ENABLE_COVERAGE=1
212
+ bundle exec rake -T ^binary:
223
213
  ```
224
214
 
225
- Once tests are completed, or during test, consult the page: [coverage/index.html](coverage/index.html)
215
+ ## Release
226
216
 
227
- ## Build
217
+ ### Branching Strategy
228
218
 
229
- By default, the gem is built signed: `make`.
230
- A private key is required to generate a signed Gem.
231
- Its path must be set using macro/envvar `SIGNING_KEY`, see below.
232
- The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
219
+ This project uses a single `main` branch for development. During the development cycle, the version in `lib/aspera/cli/version.rb` uses a `.pre` suffix (e.g., `x.y.z.pre`) to indicate a pre-release state.
233
220
 
234
- ```bash
235
- make SIGNING_KEY=/path/to/vault/gem-private_key.pem
236
- ```
221
+ Feature development and bug fixes can be done either:
237
222
 
238
- It is also possible to build an unsigned version for development purpose: `make unsigned_gem`.
223
+ - Directly on `main` for small changes
224
+ - Via feature branches with pull requests for larger changes
239
225
 
240
- ### Gem Signature
226
+ ### Checklist Before a New Release
241
227
 
242
- Refer to <certs/README.md>
228
+ When preparing for a new release, do the following:
243
229
 
244
- ### gRPC stubs for transfer SDK
230
+ - Run the test suite:
245
231
 
246
- Update with:
232
+ ```bash
233
+ bundle exec rake test:run
234
+ ```
235
+
236
+ - Verify that the container builds successfully (using the beta version):
247
237
 
248
238
  ```bash
249
- make grpc
239
+ export GEM_VERSION=$(env -u GEM_VERSION rake tools:version).$(date +%Y%m%d%H%M)
240
+ bundle exec rake container:build
241
+ bundle exec rake container:test
250
242
  ```
251
243
 
252
- It downloads the latest proto file and then compiles it.
244
+ ### Automated Release Process
253
245
 
254
- ## Container image build
246
+ Releases are triggered via the GitHub Actions UI using the **Release** workflow (`.github/workflows/release.yml`).
255
247
 
256
- See [Container build](./container/README.md).
248
+ To create a release:
257
249
 
258
- For operations, move to the folder:
250
+ 1. Navigate to **Actions** > **Release** in the GitHub repository
251
+ 2. Click **Run workflow**
252
+ 3. Optionally specify:
253
+ - **Release version**: The version to release. If left empty, uses the current version from `version.rb` without the `.pre` suffix.
254
+ - **Next development version**: The next version to prepare for. If left empty, auto-increments the minor version. The `.pre` suffix is added automatically.
255
+ 4. Click **Run workflow**
259
256
 
260
- ```bash
261
- cd container
262
- ```
257
+ The workflow automatically:
263
258
 
264
- ## Single executable build
259
+ 1. Updates `version.rb` with the release version
260
+ 2. Rebuilds documentation (PDF manual, README)
261
+ 3. Commits the changes
262
+ 4. Creates and pushes the release tag
263
+ 5. Triggers the deploy workflow to publish to [rubygems.org](https://rubygems.org/gems/aspera-cli)
264
+ 6. Updates `version.rb` to the next development version with `.pre` suffix
265
+ 7. Commits and pushes the version bump
265
266
 
266
- See [Executable build](./binary/README.md).
267
+ ### Manual Release Process (Alternative)
267
268
 
268
- For operations, move to the folder:
269
+ If needed, releases can still be done manually:
269
270
 
270
- ```bash
271
- cd binary
271
+ - Update the version in `lib/aspera/cli/version.rb` (remove `.pre` suffix)
272
+
273
+ - Build the PDF manual in `pkg`:
274
+
275
+ ```shell
276
+ bundle exec rake doc:build
277
+ ```
278
+
279
+ - Build the signed `.gem` in `pkg`:
280
+
281
+ ```shell
282
+ bundle exec rake SIGNING_KEY=/path/to/vault/gem-private_key.pem
272
283
  ```
273
284
 
274
- ## Development check list for new release
285
+ - Create the release version tag and push it to GitHub:
286
+
287
+ ```shell
288
+ bundle exec rake release_tag
289
+ ```
275
290
 
276
- When preparing for a new release do the following:
291
+ This will trigger the action `.github/workflows/deploy.yml`, which builds the gem file and pushes it to RubyGems.
277
292
 
278
- - TODO
293
+ - After release, update `version.rb` to the next development version with `.pre` suffix
279
294
 
280
- ## Long Term Implementation and delivery improvements
295
+ ## LongTerm Implementation and Delivery Improvements
281
296
 
282
- - replace Rest and OAuth classes with ruby standard gems:
297
+ - Replace custom REST and OAuth classes with standard Ruby gems:
283
298
  - <https://github.com/rest-client/rest-client>
284
299
  - <https://github.com/oauth-xx/oauth2>
285
- - use gem Thor <http://whatisthor.com/> (or other standard Ruby CLI manager)
286
- - look at <https://github.com/phusion/traveling-ruby>
300
+ - Use the `thor` gem <http://whatisthor.com/> (or other standard Ruby CLI manager)
301
+ - Look at <https://github.com/phusion/traveling-ruby>