aspera-cli 4.24.2 → 4.25.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/CHANGELOG.md +1070 -758
- data/CONTRIBUTING.md +130 -115
- data/README.md +961 -623
- data/lib/aspera/agent/direct.rb +14 -12
- data/lib/aspera/agent/factory.rb +9 -6
- data/lib/aspera/agent/transferd.rb +8 -8
- data/lib/aspera/api/aoc.rb +104 -67
- data/lib/aspera/api/ats.rb +1 -0
- data/lib/aspera/api/cos_node.rb +3 -2
- data/lib/aspera/api/faspex.rb +17 -10
- data/lib/aspera/api/node.rb +10 -12
- data/lib/aspera/ascmd.rb +2 -3
- data/lib/aspera/ascp/installation.rb +60 -46
- data/lib/aspera/ascp/management.rb +9 -5
- data/lib/aspera/assert.rb +28 -6
- data/lib/aspera/cli/error.rb +4 -2
- data/lib/aspera/cli/extended_value.rb +94 -62
- data/lib/aspera/cli/formatter.rb +44 -58
- data/lib/aspera/cli/main.rb +21 -14
- data/lib/aspera/cli/manager.rb +317 -250
- data/lib/aspera/cli/plugins/alee.rb +3 -3
- data/lib/aspera/cli/plugins/aoc.rb +139 -78
- data/lib/aspera/cli/plugins/ats.rb +30 -36
- data/lib/aspera/cli/plugins/base.rb +68 -55
- data/lib/aspera/cli/plugins/config.rb +90 -100
- data/lib/aspera/cli/plugins/console.rb +15 -9
- data/lib/aspera/cli/plugins/cos.rb +1 -1
- data/lib/aspera/cli/plugins/faspex.rb +39 -30
- data/lib/aspera/cli/plugins/faspex5.rb +57 -52
- data/lib/aspera/cli/plugins/faspio.rb +10 -7
- data/lib/aspera/cli/plugins/httpgw.rb +3 -2
- data/lib/aspera/cli/plugins/node.rb +140 -125
- data/lib/aspera/cli/plugins/oauth.rb +13 -12
- data/lib/aspera/cli/plugins/orchestrator.rb +116 -33
- data/lib/aspera/cli/plugins/preview.rb +28 -48
- data/lib/aspera/cli/plugins/server.rb +9 -10
- data/lib/aspera/cli/plugins/shares.rb +77 -43
- data/lib/aspera/cli/sync_actions.rb +49 -38
- data/lib/aspera/cli/transfer_agent.rb +16 -35
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/cli/wizard.rb +8 -5
- data/lib/aspera/command_line_builder.rb +24 -21
- data/lib/aspera/coverage.rb +6 -2
- data/lib/aspera/dot_container.rb +108 -0
- data/lib/aspera/environment.rb +71 -84
- data/lib/aspera/faspex_gw.rb +1 -1
- data/lib/aspera/faspex_postproc.rb +1 -1
- data/lib/aspera/id_generator.rb +7 -10
- data/lib/aspera/keychain/factory.rb +1 -2
- data/lib/aspera/keychain/macos_security.rb +2 -2
- data/lib/aspera/log.rb +2 -1
- data/lib/aspera/markdown.rb +31 -0
- data/lib/aspera/nagios.rb +6 -5
- data/lib/aspera/oauth/base.rb +41 -64
- data/lib/aspera/oauth/factory.rb +6 -7
- data/lib/aspera/oauth/generic.rb +1 -1
- data/lib/aspera/oauth/jwt.rb +1 -1
- data/lib/aspera/oauth/url_json.rb +6 -4
- data/lib/aspera/oauth/web.rb +2 -2
- data/lib/aspera/preview/file_types.rb +24 -38
- data/lib/aspera/preview/terminal.rb +95 -29
- data/lib/aspera/preview/utils.rb +6 -5
- data/lib/aspera/products/connect.rb +3 -3
- data/lib/aspera/rest.rb +54 -39
- data/lib/aspera/rest_error_analyzer.rb +4 -4
- data/lib/aspera/ssh.rb +10 -6
- data/lib/aspera/ssl.rb +41 -0
- data/lib/aspera/sync/conf.schema.yaml +184 -36
- data/lib/aspera/sync/database.rb +2 -1
- data/lib/aspera/sync/operations.rb +128 -72
- data/lib/aspera/transfer/parameters.rb +9 -10
- data/lib/aspera/transfer/spec.rb +2 -3
- data/lib/aspera/transfer/spec.schema.yaml +52 -22
- data/lib/aspera/transfer/spec_doc.rb +20 -30
- data/lib/aspera/uri_reader.rb +18 -4
- data/lib/transferd_pb.rb +2 -2
- data.tar.gz.sig +0 -0
- metadata +34 -6
- metadata.gz.sig +0 -0
data/CONTRIBUTING.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Contributing
|
|
2
|
-
|
|
2
|
+
|
|
3
3
|
## Reporting Issues and Vulnerabilities
|
|
4
4
|
|
|
5
|
-
If you encounter a problem or vulnerability, please report it
|
|
5
|
+
If you encounter a problem or a security vulnerability, please report it on [GitHub Issues](https://github.com/IBM/aspera-cli/issues).
|
|
6
6
|
|
|
7
7
|
Before submitting a new issue:
|
|
8
8
|
|
|
@@ -10,30 +10,32 @@ Before submitting a new issue:
|
|
|
10
10
|
|
|
11
11
|
To help us assist you efficiently, include the following in your report:
|
|
12
12
|
|
|
13
|
-
- The version of ascli you are using:
|
|
13
|
+
- The version of `ascli` you are using:
|
|
14
14
|
|
|
15
15
|
```bash
|
|
16
16
|
ascli version
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
-
- Confirmation that you
|
|
20
|
-
-
|
|
19
|
+
- Confirmation that you are using the latest version (update it if needed).
|
|
20
|
+
- Your Ruby version information:
|
|
21
21
|
|
|
22
22
|
```bash
|
|
23
|
-
ruby -
|
|
23
|
+
ruby -v
|
|
24
24
|
```
|
|
25
25
|
|
|
26
26
|
## Making Contributions
|
|
27
27
|
|
|
28
|
-
We welcome contributions to improve the aspera-cli project!
|
|
28
|
+
We welcome contributions to improve the `aspera-cli` project!
|
|
29
29
|
|
|
30
30
|
### Getting Started
|
|
31
31
|
|
|
32
|
-
Clone the repository and navigate to the project
|
|
32
|
+
Clone the repository and navigate to the project's root directory:
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
35
|
git clone https://github.com/IBM/aspera-cli.git
|
|
36
36
|
cd aspera-cli
|
|
37
|
+
bundle install
|
|
38
|
+
bundle exec rake -T
|
|
37
39
|
```
|
|
38
40
|
|
|
39
41
|
For testing instructions, refer to [Running Tests](#running-tests).
|
|
@@ -48,7 +50,7 @@ To submit a contribution:
|
|
|
48
50
|
|
|
49
51
|
1. **Implement** your feature or bug fix.
|
|
50
52
|
|
|
51
|
-
1. **Write tests** to ensure your changes are robust and
|
|
53
|
+
1. **Write tests** to ensure your changes are robust and prevent regressions.
|
|
52
54
|
|
|
53
55
|
1. Run `rubocop` to ensure your code follows the Ruby style guide.
|
|
54
56
|
|
|
@@ -61,68 +63,64 @@ To submit a contribution:
|
|
|
61
63
|
|
|
62
64
|
## Architecture
|
|
63
65
|
|
|
64
|
-
The overall architecture of aspera-cli is modular and extensible.
|
|
66
|
+
The overall architecture of `aspera-cli` is modular and extensible.
|
|
65
67
|
|
|
66
68
|

|
|
67
69
|
|
|
68
|
-
Structure Highlights
|
|
70
|
+
### Structure Highlights
|
|
69
71
|
|
|
70
72
|
- Entry Point:
|
|
71
73
|
|
|
72
|
-
`lib/aspera/cli/main.rb
|
|
74
|
+
`lib/aspera/cli/main.rb`, CLI startup logic.
|
|
73
75
|
|
|
74
76
|
- Plugins:
|
|
75
77
|
|
|
76
|
-
Located in `lib/aspera/cli/plugins
|
|
78
|
+
Located in `lib/aspera/cli/plugins`; plugins extend CLI functionality and encapsulate specific features.
|
|
77
79
|
|
|
78
80
|
- Transfer Agents:
|
|
79
81
|
|
|
80
|
-
Found in `lib/aspera/agent`, these handle data
|
|
82
|
+
Found in `lib/aspera/agent`, these handle data transfer operations.
|
|
81
83
|
|
|
82
|
-
|
|
84
|
+
Class diagrams are provided in <docs/uml.png>
|
|
83
85
|
|
|
84
|
-
## Ruby
|
|
86
|
+
## Ruby Environment
|
|
85
87
|
|
|
86
|
-
`aspera-cli` is
|
|
87
|
-
You can install Ruby using any method you prefer (e.g., rbenv
|
|
88
|
+
`aspera-cli` is written in Ruby.
|
|
89
|
+
You can install Ruby using any method you prefer (e.g., `rbenv`, `rvm`, system package manager).
|
|
88
90
|
|
|
89
|
-
To start with a clean
|
|
91
|
+
To start with a clean state and remove all installed gems:
|
|
90
92
|
|
|
91
93
|
```bash
|
|
92
|
-
|
|
94
|
+
bundle exec rake tools:clean_gems
|
|
93
95
|
```
|
|
94
96
|
|
|
95
97
|
> [!TIP]
|
|
96
98
|
> This is especially useful before testing across different Ruby versions or preparing for a release.
|
|
97
99
|
|
|
98
|
-
##
|
|
99
|
-
|
|
100
|
-
TODO: document installation of tool chain.
|
|
100
|
+
## Toolchain
|
|
101
101
|
|
|
102
|
-
|
|
102
|
+
The build system uses Ruby's `rake`.
|
|
103
103
|
|
|
104
104
|
### Environment
|
|
105
105
|
|
|
106
|
-
A few macros
|
|
106
|
+
A few macros and environment variables control certain aspects of the build:
|
|
107
107
|
|
|
108
|
-
|
|
|
109
|
-
|
|
110
|
-
| `
|
|
111
|
-
| `
|
|
112
|
-
| `
|
|
113
|
-
| `
|
|
114
|
-
| `
|
|
115
|
-
| `
|
|
116
|
-
| `SIGNING_KEY` | Path to signing key to build Gem. |
|
|
117
|
-
| `GEM_VERSION` | Gem version to build container |
|
|
108
|
+
| Environment variable | Description |
|
|
109
|
+
|-----------------------------|-----------------------------------------------------|
|
|
110
|
+
| `ASPERA_CLI_TEST_CONF_URL` | URL for configuration file with secrets for tests. |
|
|
111
|
+
| `ASPERA_CLI_DOC_CHECK_LINKS`| Check links still exist during doc generation. |
|
|
112
|
+
| `LOG_LEVEL` | Change log level in `rake` tasks. |
|
|
113
|
+
| `ENABLE_COVERAGE` | Enable test coverage analysis when set. |
|
|
114
|
+
| `SIGNING_KEY` | Path to the signing key used to build the gem file. |
|
|
115
|
+
| `GEM_VERSION` | Override gem version for builds. |
|
|
118
116
|
|
|
119
|
-
|
|
117
|
+
These can be set either as environment variables or directly on the `rake` command line.
|
|
120
118
|
|
|
121
119
|
> [!NOTE]
|
|
122
|
-
>
|
|
123
|
-
> Others are
|
|
120
|
+
> Environment variables `ASPERA_CLI_*` are typically set in the user’s shell profile for development.
|
|
121
|
+
> Others are intended for use on the command line.
|
|
124
122
|
|
|
125
|
-
To use the CLI directly from the development environment, add this to your shell profile:
|
|
123
|
+
To use the CLI directly from the development environment, add this to your shell profile (adapt the real path):
|
|
126
124
|
|
|
127
125
|
```bash
|
|
128
126
|
dev_ascli=$HOME/github/aspera-cli
|
|
@@ -134,9 +132,9 @@ export RUBYLIB=$dev_ascli/lib:$RUBYLIB
|
|
|
134
132
|
|
|
135
133
|
Documentation is generated with `pandoc` and `LaTeX`.
|
|
136
134
|
|
|
137
|
-
IBM
|
|
135
|
+
The IBM `Plex` font is used; for installation instructions, see [IBM Plex](https://www.ibm.com/plex/).
|
|
138
136
|
|
|
139
|
-
On macOS to install `lualatex` and all packages:
|
|
137
|
+
On macOS, to install `lualatex` and all packages:
|
|
140
138
|
|
|
141
139
|
```bash
|
|
142
140
|
brew install texlive
|
|
@@ -149,138 +147,155 @@ tlmgr update --self
|
|
|
149
147
|
tlmgr install fvextra selnolig lualatex-math
|
|
150
148
|
```
|
|
151
149
|
|
|
152
|
-
To check
|
|
150
|
+
To check URLs during documentation generation, set the environment variable: `ASPERA_CLI_DOC_CHECK_LINKS=1`.
|
|
151
|
+
|
|
152
|
+
To debug documentation generation, set the environment variable: `ASPERA_CLI_DOC_DEBUG=debug`.
|
|
153
153
|
|
|
154
|
-
|
|
154
|
+
## Test Environment
|
|
155
155
|
|
|
156
|
-
|
|
156
|
+
Refer to <tests/README.md>.
|
|
157
|
+
|
|
158
|
+
## Build
|
|
157
159
|
|
|
158
|
-
|
|
159
|
-
From project top folder, execute:
|
|
160
|
+
The unsigned gem is built with:
|
|
160
161
|
|
|
161
162
|
```bash
|
|
162
|
-
|
|
163
|
-
|
|
163
|
+
bundle install
|
|
164
|
+
bundle exec rake unsigned
|
|
164
165
|
```
|
|
165
166
|
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
Then, tell where this file is located:
|
|
167
|
+
If you don't want to install optional gems:
|
|
169
168
|
|
|
170
169
|
```bash
|
|
171
|
-
|
|
170
|
+
bundle config set without optional
|
|
172
171
|
```
|
|
173
172
|
|
|
174
|
-
|
|
173
|
+
To build a beta version:
|
|
175
174
|
|
|
176
175
|
```bash
|
|
177
|
-
|
|
176
|
+
export GEM_VERSION=$(env -u GEM_VERSION rake tools:version).$(date +%Y%m%d%H%M)
|
|
178
177
|
```
|
|
179
178
|
|
|
180
|
-
|
|
179
|
+
### Signed gem
|
|
181
180
|
|
|
182
|
-
|
|
181
|
+
A private key is required to generate a signed gem.
|
|
182
|
+
Its path must be set using environment variable `SIGNING_KEY`.
|
|
183
|
+
The gem is signed with the public certificate found in `certs` and the private key specified by `SIGNING_KEY` (kept secret by the maintainer).
|
|
183
184
|
|
|
184
185
|
```bash
|
|
185
|
-
|
|
186
|
-
make
|
|
187
|
-
make full
|
|
188
|
-
make list
|
|
189
|
-
make torc
|
|
190
|
-
make skip_torc
|
|
191
|
-
make skip T=orch_wizard
|
|
186
|
+
bundle exec rake SIGNING_KEY=/path/to/vault/gem-private_key.pem
|
|
192
187
|
```
|
|
193
188
|
|
|
194
|
-
|
|
189
|
+
Refer to <certs/README.md>.
|
|
195
190
|
|
|
196
|
-
|
|
197
|
-
By default, tests that use those gems are skipped.
|
|
198
|
-
To run them: `make optional`.
|
|
199
|
-
Those tests also require the optional gems to be installed: `make install_optional_gems`.
|
|
191
|
+
### gRPC stubs for transfer SDK
|
|
200
192
|
|
|
201
|
-
|
|
193
|
+
Update with:
|
|
202
194
|
|
|
203
|
-
|
|
195
|
+
```bash
|
|
196
|
+
bundle exec rake tools:grpc
|
|
197
|
+
```
|
|
204
198
|
|
|
205
|
-
|
|
199
|
+
It downloads the latest `proto` file and then compiles it.
|
|
206
200
|
|
|
207
|
-
|
|
201
|
+
## Container image build
|
|
208
202
|
|
|
209
|
-
|
|
210
|
-
2. Remove all gems: `make clean_gems`
|
|
211
|
-
3. `cd tests && make full`
|
|
203
|
+
See [Container build](./container/README.md).
|
|
212
204
|
|
|
213
|
-
|
|
205
|
+
## Single executable build
|
|
214
206
|
|
|
215
|
-
|
|
207
|
+
See [Executable build](build/binary/README.md).
|
|
216
208
|
|
|
217
|
-
|
|
218
|
-
Enable coverage monitoring using macro/envvar `ENABLE_COVERAGE`.
|
|
209
|
+
To list operations:
|
|
219
210
|
|
|
220
211
|
```bash
|
|
221
|
-
|
|
222
|
-
make ENABLE_COVERAGE=1
|
|
212
|
+
bundle exec rake -T ^binary:
|
|
223
213
|
```
|
|
224
214
|
|
|
225
|
-
|
|
215
|
+
## Release
|
|
226
216
|
|
|
227
|
-
|
|
217
|
+
### Branching Strategy
|
|
228
218
|
|
|
229
|
-
|
|
230
|
-
A private key is required to generate a signed Gem.
|
|
231
|
-
Its path must be set using macro/envvar `SIGNING_KEY`, see below.
|
|
232
|
-
The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
|
|
219
|
+
This project uses a single `main` branch for development. During the development cycle, the version in `lib/aspera/cli/version.rb` uses a `.pre` suffix (e.g., `x.y.z.pre`) to indicate a pre-release state.
|
|
233
220
|
|
|
234
|
-
|
|
235
|
-
make SIGNING_KEY=/path/to/vault/gem-private_key.pem
|
|
236
|
-
```
|
|
221
|
+
Feature development and bug fixes can be done either:
|
|
237
222
|
|
|
238
|
-
|
|
223
|
+
- Directly on `main` for small changes
|
|
224
|
+
- Via feature branches with pull requests for larger changes
|
|
239
225
|
|
|
240
|
-
###
|
|
226
|
+
### Checklist Before a New Release
|
|
241
227
|
|
|
242
|
-
|
|
228
|
+
When preparing for a new release, do the following:
|
|
243
229
|
|
|
244
|
-
|
|
230
|
+
- Run the test suite:
|
|
245
231
|
|
|
246
|
-
|
|
232
|
+
```bash
|
|
233
|
+
bundle exec rake test:run
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
- Verify that the container builds successfully (using the beta version):
|
|
247
237
|
|
|
248
238
|
```bash
|
|
249
|
-
|
|
239
|
+
export GEM_VERSION=$(env -u GEM_VERSION rake tools:version).$(date +%Y%m%d%H%M)
|
|
240
|
+
bundle exec rake container:build
|
|
241
|
+
bundle exec rake container:test
|
|
250
242
|
```
|
|
251
243
|
|
|
252
|
-
|
|
244
|
+
### Automated Release Process
|
|
253
245
|
|
|
254
|
-
|
|
246
|
+
Releases are triggered via the GitHub Actions UI using the **Release** workflow (`.github/workflows/release.yml`).
|
|
255
247
|
|
|
256
|
-
|
|
248
|
+
To create a release:
|
|
257
249
|
|
|
258
|
-
|
|
250
|
+
1. Navigate to **Actions** > **Release** in the GitHub repository
|
|
251
|
+
2. Click **Run workflow**
|
|
252
|
+
3. Optionally specify:
|
|
253
|
+
- **Release version**: The version to release. If left empty, uses the current version from `version.rb` without the `.pre` suffix.
|
|
254
|
+
- **Next development version**: The next version to prepare for. If left empty, auto-increments the minor version. The `.pre` suffix is added automatically.
|
|
255
|
+
4. Click **Run workflow**
|
|
259
256
|
|
|
260
|
-
|
|
261
|
-
cd container
|
|
262
|
-
```
|
|
257
|
+
The workflow automatically:
|
|
263
258
|
|
|
264
|
-
|
|
259
|
+
1. Updates `version.rb` with the release version
|
|
260
|
+
2. Rebuilds documentation (PDF manual, README)
|
|
261
|
+
3. Commits the changes
|
|
262
|
+
4. Creates and pushes the release tag
|
|
263
|
+
5. Triggers the deploy workflow to publish to [rubygems.org](https://rubygems.org/gems/aspera-cli)
|
|
264
|
+
6. Updates `version.rb` to the next development version with `.pre` suffix
|
|
265
|
+
7. Commits and pushes the version bump
|
|
265
266
|
|
|
266
|
-
|
|
267
|
+
### Manual Release Process (Alternative)
|
|
267
268
|
|
|
268
|
-
|
|
269
|
+
If needed, releases can still be done manually:
|
|
269
270
|
|
|
270
|
-
|
|
271
|
-
|
|
271
|
+
- Update the version in `lib/aspera/cli/version.rb` (remove `.pre` suffix)
|
|
272
|
+
|
|
273
|
+
- Build the PDF manual in `pkg`:
|
|
274
|
+
|
|
275
|
+
```shell
|
|
276
|
+
bundle exec rake doc:build
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
- Build the signed `.gem` in `pkg`:
|
|
280
|
+
|
|
281
|
+
```shell
|
|
282
|
+
bundle exec rake SIGNING_KEY=/path/to/vault/gem-private_key.pem
|
|
272
283
|
```
|
|
273
284
|
|
|
274
|
-
|
|
285
|
+
- Create the release version tag and push it to GitHub:
|
|
286
|
+
|
|
287
|
+
```shell
|
|
288
|
+
bundle exec rake release_tag
|
|
289
|
+
```
|
|
275
290
|
|
|
276
|
-
|
|
291
|
+
This will trigger the action `.github/workflows/deploy.yml`, which builds the gem file and pushes it to RubyGems.
|
|
277
292
|
|
|
278
|
-
-
|
|
293
|
+
- After release, update `version.rb` to the next development version with `.pre` suffix
|
|
279
294
|
|
|
280
|
-
## Long
|
|
295
|
+
## Long‑Term Implementation and Delivery Improvements
|
|
281
296
|
|
|
282
|
-
-
|
|
297
|
+
- Replace custom REST and OAuth classes with standard Ruby gems:
|
|
283
298
|
- <https://github.com/rest-client/rest-client>
|
|
284
299
|
- <https://github.com/oauth-xx/oauth2>
|
|
285
|
-
-
|
|
286
|
-
-
|
|
300
|
+
- Use the `thor` gem <http://whatisthor.com/> (or other standard Ruby CLI manager)
|
|
301
|
+
- Look at <https://github.com/phusion/traveling-ruby>
|