aspera-cli 4.25.3 → 4.25.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/CHANGELOG.md +18 -0
- data/CONTRIBUTING.md +98 -96
- data/lib/aspera/api/aoc.rb +5 -8
- data/lib/aspera/api/cos_node.rb +4 -0
- data/lib/aspera/api/faspex.rb +10 -1
- data/lib/aspera/api/node.rb +79 -27
- data/lib/aspera/cli/formatter.rb +27 -11
- data/lib/aspera/cli/manager.rb +13 -4
- data/lib/aspera/cli/plugins/aoc.rb +5 -3
- data/lib/aspera/cli/plugins/base.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +2 -1
- data/lib/aspera/cli/plugins/faspex5.rb +24 -12
- data/lib/aspera/cli/plugins/node.rb +7 -10
- data/lib/aspera/cli/special_values.rb +1 -0
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/cli/wizard.rb +2 -1
- data/lib/aspera/rest.rb +31 -18
- data/lib/aspera/transfer/spec.rb +1 -0
- data.tar.gz.sig +0 -0
- metadata +1 -1
- metadata.gz.sig +0 -0
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 34cb77d2badea1e3650f2aed72fb3a91c419e6457a08acd62808c9a6d9ab0e74
|
|
4
|
+
data.tar.gz: 31e1e1a28d99afebd6b6a3f7b9e848bf18256d47fdc7108368a6e964eef03b04
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e15bfb65919b8b8a6112120eca273dac962ff6d6b29d890df1e9293f4321e557bdd68697c394cb6ef738a3d5bcc3a75a2f7f177494b66542c2e970c4da99e9d0
|
|
7
|
+
data.tar.gz: 1d3570c590e7bfbda3ce5fc49396637471f48ff6fccce7235ae31adcc23c1e1ccc4a978d6bdbe9a4c1c06f06e02ed5336043267891c0b56a995b4d0199ab0d92
|
checksums.yaml.gz.sig
CHANGED
|
Binary file
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,24 @@
|
|
|
2
2
|
|
|
3
3
|
<!-- markdownlint-configure-file { "no-duplicate-heading": { "siblings_only": true } } -->
|
|
4
4
|
|
|
5
|
+
## 4.25.4
|
|
6
|
+
|
|
7
|
+
Released: 2026-03-04
|
|
8
|
+
|
|
9
|
+
### New Features
|
|
10
|
+
|
|
11
|
+
* **general**: If `@:` is used, then marker `END` optionally marks the end of collected arguments.
|
|
12
|
+
* `format`: `display` defaults to `info` only if `format` is set to `table`, else defaults to `data`.
|
|
13
|
+
* `node`: Parameter `accept_v4` of option `node_api` (boolean, defaults to `true`) allows using gen4 browsing with `Accept-Version: 4.0` for best performance when there are thousands of files.
|
|
14
|
+
|
|
15
|
+
### Issues Fixed
|
|
16
|
+
|
|
17
|
+
* `faspex5`: Listing content, or receiving a package requires API parameter: `recipient_user_id` or `recipient_workgroup_id`, else error `Not authorized` is returned.
|
|
18
|
+
|
|
19
|
+
### Breaking Changes
|
|
20
|
+
|
|
21
|
+
* `node`: Options `node_cache` and `default_ports` are replaced with option: `node_api` (`Hash`) with boolean parameters (keys): `cache` and `standard_ports` with default value `true`.
|
|
22
|
+
|
|
5
23
|
## 4.25.3
|
|
6
24
|
|
|
7
25
|
Released: 2026-02-18
|
data/CONTRIBUTING.md
CHANGED
|
@@ -2,22 +2,23 @@
|
|
|
2
2
|
|
|
3
3
|
## Reporting Issues and Vulnerabilities
|
|
4
4
|
|
|
5
|
-
If you encounter a
|
|
5
|
+
If you encounter a bug or a security vulnerability, please report it via [GitHub Issues](https://github.com/IBM/aspera-cli/issues).
|
|
6
6
|
|
|
7
|
-
Before submitting a new
|
|
7
|
+
Before submitting a new report:
|
|
8
8
|
|
|
9
|
-
- Search existing issues to
|
|
9
|
+
- **Search existing issues** to determine if the problem has already been documented or resolved.
|
|
10
10
|
|
|
11
|
-
To help us
|
|
11
|
+
To help us diagnose and resolve the issue efficiently, please include the following in your report:
|
|
12
12
|
|
|
13
|
-
- The
|
|
13
|
+
- The `ascli` version you are using:
|
|
14
14
|
|
|
15
15
|
```bash
|
|
16
|
-
ascli
|
|
16
|
+
ascli -v
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
-
-
|
|
20
|
-
|
|
19
|
+
- **Update confirmation**: Verify that you are running the latest version.
|
|
20
|
+
|
|
21
|
+
- **Your Ruby environment details**:
|
|
21
22
|
|
|
22
23
|
```bash
|
|
23
24
|
ruby -v
|
|
@@ -29,7 +30,7 @@ We welcome contributions to improve the `aspera-cli` project!
|
|
|
29
30
|
|
|
30
31
|
### Getting Started
|
|
31
32
|
|
|
32
|
-
Clone the repository
|
|
33
|
+
Clone the repository to initialize the development environment:
|
|
33
34
|
|
|
34
35
|
```bash
|
|
35
36
|
git clone https://github.com/IBM/aspera-cli.git
|
|
@@ -38,55 +39,55 @@ bundle install
|
|
|
38
39
|
bundle exec rake -T
|
|
39
40
|
```
|
|
40
41
|
|
|
41
|
-
For testing instructions, refer to [Running Tests](#running-tests).
|
|
42
|
+
For detailed testing instructions, please refer to [Running Tests](#running-tests).
|
|
42
43
|
|
|
43
44
|
### How to Contribute
|
|
44
45
|
|
|
45
|
-
To submit a contribution:
|
|
46
|
+
To submit a contribution, follow these steps:
|
|
46
47
|
|
|
47
48
|
1. **Fork** the repository on GitHub.
|
|
48
49
|
|
|
49
|
-
1. **Create a feature branch** for your changes.
|
|
50
|
+
1. **Create a feature branch** specifically for your changes.
|
|
50
51
|
|
|
51
52
|
1. **Implement** your feature or bug fix.
|
|
52
53
|
|
|
53
54
|
1. **Write tests** to ensure your changes are robust and prevent regressions.
|
|
54
55
|
|
|
55
|
-
1. Run `rubocop` to ensure your code
|
|
56
|
+
1. **Run** `rubocop` to ensure your code adheres to the Ruby style guide.
|
|
56
57
|
|
|
57
|
-
1. Update `CHANGELOG.md` with a summary of your changes.
|
|
58
|
+
1. **Update** `CHANGELOG.md` with a concise summary of your changes.
|
|
58
59
|
|
|
59
|
-
1. Submit a
|
|
60
|
+
1. **Submit a pull request** with a detailed description of your work.
|
|
60
61
|
|
|
61
62
|
> [!TIP]
|
|
62
|
-
>
|
|
63
|
+
> Keep pull requests focused; include only changes relevant to the specific feature or fix.
|
|
63
64
|
|
|
64
65
|
## Architecture
|
|
65
66
|
|
|
66
|
-
The
|
|
67
|
+
The `aspera-cli` architecture is designed to be modular and extensible.
|
|
67
68
|
|
|
68
69
|

|
|
69
70
|
|
|
70
71
|
### Structure Highlights
|
|
71
72
|
|
|
72
|
-
- Entry Point
|
|
73
|
+
- **Entry Point**:
|
|
73
74
|
|
|
74
|
-
`lib/aspera/cli/main.rb
|
|
75
|
+
`lib/aspera/cli/main.rb` contains the core CLI startup logic.
|
|
75
76
|
|
|
76
|
-
- Plugins
|
|
77
|
+
- **Plugins**:
|
|
77
78
|
|
|
78
|
-
Located in `lib/aspera/cli/plugins
|
|
79
|
+
Located in `lib/aspera/cli/plugins`, these extend CLI functionality and encapsulate specific features.
|
|
79
80
|
|
|
80
|
-
- Transfer Agents
|
|
81
|
+
- **Transfer Agents**:
|
|
81
82
|
|
|
82
|
-
|
|
83
|
+
Located in `lib/aspera/agent`, these components manage data transfer operations.
|
|
83
84
|
|
|
84
|
-
|
|
85
|
+
Detailed class diagrams are available in <docs/uml.png>
|
|
85
86
|
|
|
86
87
|
## Ruby Environment
|
|
87
88
|
|
|
88
|
-
`aspera-cli` is
|
|
89
|
-
You can
|
|
89
|
+
`aspera-cli` is built with Ruby.
|
|
90
|
+
You can manage your Ruby installation using your preferred tool (e.g., `rbenv`, `rvm`, or a system package manager).
|
|
90
91
|
|
|
91
92
|
To start with a clean state and remove all installed gems:
|
|
92
93
|
|
|
@@ -95,35 +96,34 @@ bundle exec rake tools:clean_gems
|
|
|
95
96
|
```
|
|
96
97
|
|
|
97
98
|
> [!TIP]
|
|
98
|
-
> This is
|
|
99
|
+
> This is particularly useful when testing across different Ruby versions or preparing for a new release.
|
|
99
100
|
|
|
100
101
|
## Toolchain
|
|
101
102
|
|
|
102
|
-
The build system
|
|
103
|
+
The build system is powered by Ruby's `rake`.
|
|
103
104
|
|
|
104
|
-
### Environment
|
|
105
|
+
### Environment Configuration
|
|
105
106
|
|
|
106
|
-
|
|
107
|
+
The following environment variables and macros control specific build behaviors:
|
|
107
108
|
|
|
108
|
-
| Environment variable | Description
|
|
109
|
-
|
|
110
|
-
| `ASPERA_CLI_TEST_CONF_URL` | URL for configuration file
|
|
111
|
-
| `ASPERA_CLI_DOC_CHECK_LINKS`|
|
|
112
|
-
| `
|
|
113
|
-
| `
|
|
114
|
-
| `ENABLE_COVERAGE` |
|
|
115
|
-
| `SIGNING_KEY` | Path to the signing key used
|
|
116
|
-
| `SIGNING_KEY_PEM` | PEM of signing key.
|
|
109
|
+
| Environment variable | Contents | Description |
|
|
110
|
+
|-----------------------------|------------| -------------------------------------------------------------|
|
|
111
|
+
| `ASPERA_CLI_TEST_CONF_URL` | URL | URL for the configuration file containing secrets for tests. |
|
|
112
|
+
| `ASPERA_CLI_DOC_CHECK_LINKS`| yes/no | Validates that links exist during documentation generation. |
|
|
113
|
+
| `LOG_SECRETS` | yes/no | Toggles the logging of secrets in `rake` tasks. |
|
|
114
|
+
| `LOG_LEVEL` | debug, ... | Sets the logging verbosity for `rake` tasks. |
|
|
115
|
+
| `ENABLE_COVERAGE` | set/unset | Enables test coverage analysis when defined. |
|
|
116
|
+
| `SIGNING_KEY` | File path | Path to the signing key used for building the gem file. |
|
|
117
|
+
| `SIGNING_KEY_PEM` | PEM Value | The PEM content of the signing key. |
|
|
117
118
|
|
|
118
|
-
These can be set
|
|
119
|
+
These values can be set as standard environment variables or passed directly to the `rake` command.
|
|
119
120
|
|
|
120
|
-
Setting `SIGNING_KEY_PEM`
|
|
121
|
+
Setting `SIGNING_KEY_PEM` automatically generates a file at `$HOME/.gem/signing_key.pem` and sets the `SIGNING_KEY` variable accordingly.
|
|
121
122
|
|
|
122
123
|
> [!NOTE]
|
|
123
|
-
>
|
|
124
|
-
> Others are intended for use on the command line.
|
|
124
|
+
> `ASPERA_CLI_*` variables are typically defined in your shell profile for development, while others are intended for ad-hoc command-line use.
|
|
125
125
|
|
|
126
|
-
To
|
|
126
|
+
To run the CLI directly from your source directory, add the following to your shell profile (adjust the path as necessary):
|
|
127
127
|
|
|
128
128
|
```bash
|
|
129
129
|
dev_ascli=$HOME/github/aspera-cli
|
|
@@ -135,26 +135,27 @@ export RUBYLIB=$dev_ascli/lib:$RUBYLIB
|
|
|
135
135
|
|
|
136
136
|
Documentation is generated with `pandoc` and `LaTeX`.
|
|
137
137
|
|
|
138
|
-
The
|
|
138
|
+
The project utilizes the **IBM Plex font**.
|
|
139
|
+
Installation instructions can be found at [IBM Plex](https://www.ibm.com/plex/).
|
|
139
140
|
|
|
140
|
-
On macOS,
|
|
141
|
+
On macOS, install `lualatex` and required packages via Homebrew:
|
|
141
142
|
|
|
142
143
|
```bash
|
|
143
144
|
brew install texlive
|
|
144
145
|
```
|
|
145
146
|
|
|
146
|
-
If
|
|
147
|
+
If using an alternative installation method, ensure the following packages are present:
|
|
147
148
|
|
|
148
149
|
```bash
|
|
149
150
|
tlmgr update --self
|
|
150
151
|
tlmgr install fvextra selnolig lualatex-math
|
|
151
152
|
```
|
|
152
153
|
|
|
153
|
-
To
|
|
154
|
+
- To validate URLs during generation: `ASPERA_CLI_DOC_CHECK_LINKS=1`.
|
|
154
155
|
|
|
155
|
-
To debug
|
|
156
|
+
- To debug the generation process: `ASPERA_CLI_DOC_DEBUG=debug`.
|
|
156
157
|
|
|
157
|
-
To
|
|
158
|
+
- To build the documentation:
|
|
158
159
|
|
|
159
160
|
```bash
|
|
160
161
|
rake doc:build
|
|
@@ -162,18 +163,18 @@ rake doc:build
|
|
|
162
163
|
|
|
163
164
|
## Test Environment
|
|
164
165
|
|
|
165
|
-
|
|
166
|
+
Detailed testing information can be found in <tests/README.md>.
|
|
166
167
|
|
|
167
168
|
## Build
|
|
168
169
|
|
|
169
|
-
|
|
170
|
+
To build an unsigned gem:
|
|
170
171
|
|
|
171
172
|
```bash
|
|
172
173
|
bundle install
|
|
173
174
|
bundle exec rake unsigned
|
|
174
175
|
```
|
|
175
176
|
|
|
176
|
-
|
|
177
|
+
To exclude optional gems from the installation:
|
|
177
178
|
|
|
178
179
|
```bash
|
|
179
180
|
bundle config set without optional
|
|
@@ -181,35 +182,34 @@ bundle config set without optional
|
|
|
181
182
|
|
|
182
183
|
### Signed gem
|
|
183
184
|
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
The gem is signed with the public certificate found in `certs` and the private key specified by `SIGNING_KEY` (kept secret by the maintainer).
|
|
185
|
+
Generating a signed gem requires a **private key**, specified via the `SIGNING_KEY` environment variable.
|
|
186
|
+
The gem is signed using the public certificate in `certs` and the **private key**.
|
|
187
187
|
|
|
188
188
|
```bash
|
|
189
189
|
bundle exec rake SIGNING_KEY=/path/to/vault/gem-private_key.pem
|
|
190
190
|
```
|
|
191
191
|
|
|
192
|
-
|
|
192
|
+
For more details, see <certs/README.md>.
|
|
193
193
|
|
|
194
|
-
### gRPC stubs for
|
|
194
|
+
### gRPC stubs for Transfer SDK
|
|
195
195
|
|
|
196
|
-
|
|
196
|
+
To update the stubs:
|
|
197
197
|
|
|
198
198
|
```bash
|
|
199
199
|
bundle exec rake tools:grpc
|
|
200
200
|
```
|
|
201
201
|
|
|
202
|
-
|
|
202
|
+
This task downloads the latest `.proto` files and compiles them into the Ruby source files included in the repository.
|
|
203
203
|
|
|
204
204
|
## Container image build
|
|
205
205
|
|
|
206
|
-
|
|
206
|
+
Refer to the [Container build guide](./container/README.md).
|
|
207
207
|
|
|
208
208
|
## Single executable build
|
|
209
209
|
|
|
210
|
-
|
|
210
|
+
Refer to the [Executable build guide](build/binary/README.md).
|
|
211
211
|
|
|
212
|
-
To list
|
|
212
|
+
To list related `rake` tasks:
|
|
213
213
|
|
|
214
214
|
```bash
|
|
215
215
|
bundle exec rake -T ^binary:
|
|
@@ -219,25 +219,26 @@ bundle exec rake -T ^binary:
|
|
|
219
219
|
|
|
220
220
|
### Branching Strategy
|
|
221
221
|
|
|
222
|
-
This project
|
|
223
|
-
During
|
|
222
|
+
This project maintains a single `main` branch.
|
|
223
|
+
During development, the version in `lib/aspera/cli/version.rb` includes a `.pre` suffix (e.g., `x.y.z.pre`).
|
|
224
|
+
|
|
225
|
+
Contributions are handled as follows:
|
|
224
226
|
|
|
225
|
-
|
|
227
|
+
- **Direct commits** to `main`: Permitted for minor changes.
|
|
226
228
|
|
|
227
|
-
-
|
|
228
|
-
- Via feature branches with pull requests for larger changes
|
|
229
|
+
- **Feature branches**: Required for significant changes via pull requests.
|
|
229
230
|
|
|
230
|
-
### Checklist
|
|
231
|
+
### Pre-Release Checklist
|
|
231
232
|
|
|
232
|
-
|
|
233
|
+
Before a new release, ensure the following:
|
|
233
234
|
|
|
234
|
-
-
|
|
235
|
+
- **Pass all tests**:
|
|
235
236
|
|
|
236
237
|
```bash
|
|
237
238
|
bundle exec rake test:run
|
|
238
239
|
```
|
|
239
240
|
|
|
240
|
-
- Verify
|
|
241
|
+
- **Verify container builds** (using the local gem):
|
|
241
242
|
|
|
242
243
|
```bash
|
|
243
244
|
bundle exec rake container:build'[local]'
|
|
@@ -246,60 +247,61 @@ bundle exec rake container:test
|
|
|
246
247
|
|
|
247
248
|
### Automated Release Process
|
|
248
249
|
|
|
249
|
-
Releases are
|
|
250
|
+
Releases are managed through the GitHub Actions UI via the **New Release on GitHub** workflow (`.github/workflows/release.yml`).
|
|
251
|
+
|
|
252
|
+
1. Navigate to **Actions** > **New Release on GitHub**
|
|
253
|
+
2. Select **Run workflow**
|
|
254
|
+
3. (Optionally) Specify:
|
|
255
|
+
- **Release version**: Defaults to the current `version.rb` value (minus the `.pre` suffix).
|
|
250
256
|
|
|
251
|
-
|
|
257
|
+
e.g. current `a.b.c.pre` → `a.b.c`.
|
|
258
|
+
- **Next development version**: Defaults to an incremented minor version with the `.pre` suffix.
|
|
252
259
|
|
|
253
|
-
|
|
254
|
-
2. Click **Run workflow**
|
|
255
|
-
3. Optionally specify:
|
|
256
|
-
- **Release version**: The version to release. If left empty, uses the current version from `version.rb` without the `.pre` suffix.
|
|
257
|
-
- **Next development version**: The next version to prepare for. If left empty, auto-increments the minor version. The `.pre` suffix is added automatically.
|
|
260
|
+
e.g. release `a.b.c` → `a.(b+1).0.pre`.
|
|
258
261
|
4. Click **Run workflow**
|
|
259
262
|
|
|
260
|
-
The workflow
|
|
263
|
+
The automated workflow performs the following:
|
|
261
264
|
|
|
262
|
-
1. Updates `version.rb`
|
|
263
|
-
2. Rebuilds documentation (PDF
|
|
265
|
+
1. Updates `version.rb` to the release version
|
|
266
|
+
2. Rebuilds all documentation (PDF and Markdown)
|
|
264
267
|
3. Commits the changes
|
|
265
268
|
4. Creates and pushes the release tag
|
|
266
269
|
5. Triggers the `deploy` workflow to publish to [rubygems.org](https://rubygems.org/gems/aspera-cli)
|
|
267
|
-
6.
|
|
268
|
-
7. Commits and pushes the version bump
|
|
270
|
+
6. Increments `version.rb` to the next development version.
|
|
271
|
+
7. Commits and pushes the version bump to `main`.
|
|
269
272
|
|
|
270
273
|
### Manual Release Process (Alternative)
|
|
271
274
|
|
|
272
|
-
If
|
|
273
|
-
Basically, follow the same procedure as in the GitHub action:
|
|
275
|
+
If necessary, you can mirror the automated process manually:
|
|
274
276
|
|
|
275
277
|
- Update the version in `lib/aspera/cli/version.rb` (remove `.pre` suffix)
|
|
276
278
|
|
|
277
|
-
- Build the PDF manual
|
|
279
|
+
- Build the PDF manual:
|
|
278
280
|
|
|
279
281
|
```shell
|
|
280
282
|
bundle exec rake doc:build
|
|
281
283
|
```
|
|
282
284
|
|
|
283
|
-
- Build the signed
|
|
285
|
+
- Build the signed gem:
|
|
284
286
|
|
|
285
287
|
```shell
|
|
286
288
|
bundle exec rake SIGNING_KEY=/path/to/vault/gem-private_key.pem
|
|
287
289
|
```
|
|
288
290
|
|
|
289
|
-
-
|
|
291
|
+
- Tag the release and push to GitHub:
|
|
290
292
|
|
|
291
293
|
```shell
|
|
292
294
|
bundle exec rake release_tag
|
|
293
295
|
```
|
|
294
296
|
|
|
295
|
-
This
|
|
297
|
+
This triggers the `.github/workflows/deploy.yml` action to publish to RubyGems.
|
|
296
298
|
|
|
297
|
-
-
|
|
299
|
+
- Update `version.rb` to the next `.pre` development version.
|
|
298
300
|
|
|
299
301
|
## Future Improvements
|
|
300
302
|
|
|
301
|
-
-
|
|
302
|
-
-
|
|
303
|
-
-
|
|
304
|
-
-
|
|
305
|
-
-
|
|
303
|
+
- Evaluate replacing custom REST and OAuth implementations with standard gems:
|
|
304
|
+
- [rest-client](https://github.com/rest-client/rest-client)
|
|
305
|
+
- [oauth2](https://github.com/oauth-xx/oauth2)
|
|
306
|
+
- Integrate `thor` <http://whatisthor.com/> or another standard Ruby CLI framework.
|
|
307
|
+
- Explore [Traveling Ruby](https://github.com/phusion/traveling-ruby) for distribution.
|
data/lib/aspera/api/aoc.rb
CHANGED
|
@@ -145,11 +145,10 @@ module Aspera
|
|
|
145
145
|
# Call `block` with same query using paging and response information.
|
|
146
146
|
# Block must return a 2 element `Array` with data and http response
|
|
147
147
|
# @param query [Hash] Additionnal query parameters
|
|
148
|
-
# @param progress [nil, Object] Uses methods: `long_operation_running` and `long_operation_terminated`
|
|
149
148
|
# @return [Hash] Items and total number of items
|
|
150
149
|
# @option return [Array<Hash>] :items The list of items
|
|
151
150
|
# @option return [Integer] :total The total number of items
|
|
152
|
-
def call_paging(query: {}
|
|
151
|
+
def call_paging(query: {})
|
|
153
152
|
Aspera.assert_type(query, Hash){'query'}
|
|
154
153
|
Aspera.assert(block_given?)
|
|
155
154
|
# set default large page if user does not specify own parameters. AoC Caps to 1000 anyway
|
|
@@ -177,9 +176,9 @@ module Aspera
|
|
|
177
176
|
break if !max_items.nil? && item_list.count >= max_items
|
|
178
177
|
break if !max_pages.nil? && page_count >= max_pages
|
|
179
178
|
break if total_count&.<=(item_list.count)
|
|
180
|
-
|
|
179
|
+
RestParameters.instance.spinner_cb.call("#{item_list.count} / #{total_count}") unless total_count.eql?(item_list.count.to_s)
|
|
181
180
|
end
|
|
182
|
-
|
|
181
|
+
RestParameters.instance.spinner_cb.call(action: :success)
|
|
183
182
|
item_list = item_list[0..max_items - 1] if !max_items.nil? && item_list.count > max_items
|
|
184
183
|
return {items: item_list, total: total_count}
|
|
185
184
|
end
|
|
@@ -232,8 +231,7 @@ module Aspera
|
|
|
232
231
|
username: nil,
|
|
233
232
|
password: nil,
|
|
234
233
|
workspace: nil,
|
|
235
|
-
secret_finder: nil
|
|
236
|
-
progress_disp: nil
|
|
234
|
+
secret_finder: nil
|
|
237
235
|
)
|
|
238
236
|
# Test here because link may set url
|
|
239
237
|
Aspera.assert(url, 'Missing mandatory option: url', type: ParameterError)
|
|
@@ -244,7 +242,6 @@ module Aspera
|
|
|
244
242
|
# key: access key
|
|
245
243
|
# value: associated secret
|
|
246
244
|
@secret_finder = secret_finder
|
|
247
|
-
@progress_disp = progress_disp
|
|
248
245
|
@workspace_name = workspace
|
|
249
246
|
@cache_user_info = nil
|
|
250
247
|
@cache_url_token_info = nil
|
|
@@ -309,7 +306,7 @@ module Aspera
|
|
|
309
306
|
# @param query [nil, Hash] Additional query
|
|
310
307
|
# @return [Hash] {items: , total: }
|
|
311
308
|
def read_with_paging(subpath, query = nil)
|
|
312
|
-
return self.class.call_paging(query: query
|
|
309
|
+
return self.class.call_paging(query: query) do |paged_query|
|
|
313
310
|
read(subpath, query: paged_query, ret: :both)
|
|
314
311
|
end
|
|
315
312
|
end
|
data/lib/aspera/api/cos_node.rb
CHANGED
|
@@ -10,6 +10,7 @@ module Aspera
|
|
|
10
10
|
class CosNode < Node
|
|
11
11
|
IBM_CLOUD_TOKEN_URL = 'https://iam.cloud.ibm.com/identity'
|
|
12
12
|
TOKEN_FIELD = 'delegated_refresh_token'
|
|
13
|
+
FASP_INFO_KEYS = %w[ATSEndpoint AccessKey].freeze
|
|
13
14
|
class << self
|
|
14
15
|
def parameters_from_svc_credentials(service_credentials, bucket_region)
|
|
15
16
|
# check necessary contents
|
|
@@ -60,6 +61,9 @@ module Aspera
|
|
|
60
61
|
).body
|
|
61
62
|
ats_info = XmlSimple.xml_in(xml_result_text, {'ForceArray' => false})
|
|
62
63
|
Log.dump(:ats_info, ats_info)
|
|
64
|
+
Aspera.assert_hash_all(ats_info, String, nil){'ats_info'}
|
|
65
|
+
Aspera.assert((FASP_INFO_KEYS - ats_info.keys).empty?){'ats_info missing required keys'}
|
|
66
|
+
Aspera.assert_hash_all(ats_info['AccessKey'], String, String){'ats_info'}
|
|
63
67
|
@storage_credentials = {
|
|
64
68
|
'type' => 'token',
|
|
65
69
|
'token' => {TOKEN_FIELD => nil}
|
data/lib/aspera/api/faspex.rb
CHANGED
|
@@ -91,7 +91,7 @@ module Aspera
|
|
|
91
91
|
JOB_RUNNING = %w[queued working].freeze
|
|
92
92
|
PATH_STANDARD_ROOT = '/aspera/faspex'
|
|
93
93
|
PATH_API_DETECT = "#{PATH_API_V5}/#{PATH_HEALTH}"
|
|
94
|
-
|
|
94
|
+
HEADER_X_NEXT_ITER_TOKEN = 'X-Aspera-Next-Iteration-Token'
|
|
95
95
|
HEADER_FASPEX_VERSION = 'X-IBM-Aspera'
|
|
96
96
|
EMAIL_NOTIF_LIST = %w[
|
|
97
97
|
welcome_email
|
|
@@ -136,6 +136,15 @@ module Aspera
|
|
|
136
136
|
end
|
|
137
137
|
attr_reader :pub_link_context
|
|
138
138
|
|
|
139
|
+
# @param url Faspex URL, can be a public link
|
|
140
|
+
# @param auth Authentication method: :boot (token in header), :web (open browser), :jwt (client_id + private key), :public_link (context in URL)
|
|
141
|
+
# @param password For :boot auth, the token copied directly from browser in developer mode
|
|
142
|
+
# @param client_id For :web and :jwt auth, the client_id of web UI application
|
|
143
|
+
# @param client_secret For :web auth, the client_secret of web UI application (not needed for :jwt)
|
|
144
|
+
# @param redirect_uri For :web auth, the redirect_uri of web UI application (must be the same as in application configuration)
|
|
145
|
+
# @param username For :jwt auth, the username of the user to impersonate
|
|
146
|
+
# @param private_key For :jwt auth, the private key to sign JWT token
|
|
147
|
+
# @param passphrase For :jwt auth, the passphrase of the private key
|
|
139
148
|
def initialize(
|
|
140
149
|
url:,
|
|
141
150
|
auth:,
|
data/lib/aspera/api/node.rb
CHANGED
|
@@ -6,16 +6,16 @@ require 'aspera/oauth'
|
|
|
6
6
|
require 'aspera/log'
|
|
7
7
|
require 'aspera/assert'
|
|
8
8
|
require 'aspera/environment'
|
|
9
|
-
require 'zlib'
|
|
10
9
|
require 'base64'
|
|
11
10
|
require 'openssl'
|
|
12
11
|
require 'pathname'
|
|
12
|
+
require 'zlib'
|
|
13
13
|
require 'net/ssh/buffer'
|
|
14
14
|
|
|
15
15
|
module Aspera
|
|
16
16
|
module Api
|
|
17
17
|
# Provides additional functions using node API with gen4 extensions (access keys)
|
|
18
|
-
class Node <
|
|
18
|
+
class Node < Rest
|
|
19
19
|
# Format of node scope : node.<access key>:<scope>
|
|
20
20
|
module Scope
|
|
21
21
|
# Node sub-scopes
|
|
@@ -42,9 +42,10 @@ module Aspera
|
|
|
42
42
|
ACCESS_LEVELS = %w[delete list mkdir preview read rename write].freeze
|
|
43
43
|
# Special HTTP Headers
|
|
44
44
|
HEADER_X_ASPERA_ACCESS_KEY = 'X-Aspera-AccessKey'
|
|
45
|
-
HEADER_X_TOTAL_COUNT = 'X-Total-Count'
|
|
46
45
|
HEADER_X_CACHE_CONTROL = 'X-Aspera-Cache-Control'
|
|
46
|
+
HEADER_X_TOTAL_COUNT = 'X-Total-Count'
|
|
47
47
|
HEADER_X_NEXT_ITER_TOKEN = 'X-Aspera-Next-Iteration-Token'
|
|
48
|
+
HEADER_ACCEPT_VERSION = 'Accept-Version'
|
|
48
49
|
# / in cloud
|
|
49
50
|
PATH_SEPARATOR = '/'
|
|
50
51
|
|
|
@@ -52,22 +53,36 @@ module Aspera
|
|
|
52
53
|
OAuth::Factory.instance.register_decoder(lambda{ |token| Node.decode_bearer_token(token)})
|
|
53
54
|
|
|
54
55
|
# Class instance variable, access with accessors on class
|
|
55
|
-
@
|
|
56
|
-
@use_node_cache = true
|
|
57
|
-
|
|
58
|
-
class << self
|
|
56
|
+
@api_options = {
|
|
59
57
|
# Set to false to read transfer parameters from download_setup
|
|
60
|
-
|
|
58
|
+
standard_ports: true,
|
|
61
59
|
# Set to false to bypass cache in redis
|
|
62
|
-
|
|
60
|
+
cache: true,
|
|
61
|
+
accept_v4: true
|
|
62
|
+
}
|
|
63
|
+
OPTIONS = @api_options.keys.freeze
|
|
64
|
+
|
|
65
|
+
class << self
|
|
66
|
+
attr_reader :api_options
|
|
63
67
|
attr_reader :use_dynamic_key
|
|
64
68
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
69
|
+
def api_options=(h)
|
|
70
|
+
Aspera.assert_type(h, Hash)
|
|
71
|
+
h.each do |k, v|
|
|
72
|
+
Aspera.assert(@api_options.key?(k.to_sym)){"unknown api option: #{k} (#{OPTIONS.join(', ')})"}
|
|
73
|
+
Aspera.assert_type(v, TrueClass, FalseClass){"api options value for #{k} should be boolean"}
|
|
74
|
+
@api_options[k.to_sym] = v
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# Adds cache control header for node API /files/:id
|
|
79
|
+
# as globally specified to read request
|
|
80
|
+
# Use like this: read(..., headers: add_cache_control)
|
|
81
|
+
# @param headers [Hash] optional initial headers to add to
|
|
82
|
+
# @return [Hash] headers with cache control header added if needed
|
|
83
|
+
def add_cache_control(headers = {})
|
|
84
|
+
headers[HEADER_X_CACHE_CONTROL] = 'no-cache' unless api_options[:cache]
|
|
85
|
+
headers
|
|
71
86
|
end
|
|
72
87
|
|
|
73
88
|
# Set private key to be used
|
|
@@ -180,8 +195,8 @@ module Aspera
|
|
|
180
195
|
Aspera.assert(!access_key.nil?)
|
|
181
196
|
end
|
|
182
197
|
return {
|
|
183
|
-
|
|
184
|
-
'Authorization'
|
|
198
|
+
HEADER_X_ASPERA_ACCESS_KEY => access_key,
|
|
199
|
+
'Authorization' => bearer_auth
|
|
185
200
|
}
|
|
186
201
|
end
|
|
187
202
|
end
|
|
@@ -248,6 +263,50 @@ module Aspera
|
|
|
248
263
|
return false
|
|
249
264
|
end
|
|
250
265
|
|
|
266
|
+
# Read folder content, with pagination management for gen4, not recursive
|
|
267
|
+
# if `Accept-Version: 4.0` is not specified:
|
|
268
|
+
# if `page` and `per_page` are not specified, then all entries are returned.
|
|
269
|
+
# if either `page` or `per_page` is specified, then both are required, else 400
|
|
270
|
+
# if `Accept-Version: 4.0` is specified:
|
|
271
|
+
# those queries are not available: page (not mentioned), sort, min_size, max_size, min_modified_time, max_modified_time, target_id, target_node_id, files_prefetch_count, page, name_iglob : either ignored or result in API error 400.
|
|
272
|
+
# query include is accepted, but seems to do nothing as access_levels and recursive_counts are already included in results.
|
|
273
|
+
# query `iteration_token` is accepted and allows to get paginated results, with `X-Aspera-Next-Iteration-Token` header in response to get next page token. `X-Aspera-Total-Count` header gives total count of entries.
|
|
274
|
+
def read_folder_content(file_id, query = nil, exception: true, path: nil)
|
|
275
|
+
folder_items = []
|
|
276
|
+
begin
|
|
277
|
+
query ||= {}
|
|
278
|
+
headers = self.class.add_cache_control
|
|
279
|
+
use_v4 = self.class.api_options[:accept_v4]
|
|
280
|
+
return read("files/#{file_id}/files", query, headers: headers) unless use_v4 || query.key?('page') || query.key?('per_page')
|
|
281
|
+
if use_v4
|
|
282
|
+
headers[HEADER_ACCEPT_VERSION] = '4.0'
|
|
283
|
+
query['per_page'] = 1000 unless query.key?('per_page')
|
|
284
|
+
elsif query.key?('per_page') && !query.key?('page')
|
|
285
|
+
query['page'] = 0
|
|
286
|
+
end
|
|
287
|
+
loop do
|
|
288
|
+
RestParameters.instance.spinner_cb.call(folder_items.count)
|
|
289
|
+
data, http = read("files/#{file_id}/files", query, headers: headers, ret: :both)
|
|
290
|
+
folder_items.concat(data)
|
|
291
|
+
if use_v4
|
|
292
|
+
iteration_token = http[HEADER_X_NEXT_ITER_TOKEN]
|
|
293
|
+
break if iteration_token.nil? || iteration_token.empty?
|
|
294
|
+
query['iteration_token'] = iteration_token
|
|
295
|
+
else
|
|
296
|
+
break if data['item_count'].eql?(0)
|
|
297
|
+
query['offset'] += data['item_count']
|
|
298
|
+
end
|
|
299
|
+
end
|
|
300
|
+
rescue StandardError => e
|
|
301
|
+
raise e if exception
|
|
302
|
+
Log.log.warn{"#{path}: #{e.class} #{e.message}"}
|
|
303
|
+
Log.log.debug{(['Backtrace:'] + e.backtrace).join("\n")}
|
|
304
|
+
ensure
|
|
305
|
+
RestParameters.instance.spinner_cb.call(folder_items.count, action: :success)
|
|
306
|
+
end
|
|
307
|
+
folder_items
|
|
308
|
+
end
|
|
309
|
+
|
|
251
310
|
# Recursively browse in a folder (with non-recursive method)
|
|
252
311
|
# Entries of folders are processed if the processing method returns true
|
|
253
312
|
# Links are processed on the respective node
|
|
@@ -267,14 +326,7 @@ module Aspera
|
|
|
267
326
|
current_item = folders_to_explore.shift
|
|
268
327
|
Log.log.debug{"Exploring #{current_item[:path]}".bg_green}
|
|
269
328
|
# Get folder content
|
|
270
|
-
folder_contents =
|
|
271
|
-
begin
|
|
272
|
-
# TODO: use header
|
|
273
|
-
read("files/#{current_item[:id]}/files", query, **self.class.cache_control)
|
|
274
|
-
rescue StandardError => e
|
|
275
|
-
Log.log.warn{"#{current_item[:path]}: #{e.class} #{e.message}"}
|
|
276
|
-
[]
|
|
277
|
-
end
|
|
329
|
+
folder_contents = read_folder_content(current_item[:id], query, exception: false, path: current_item[:path])
|
|
278
330
|
Log.dump(:folder_contents, folder_contents)
|
|
279
331
|
folder_contents.each do |entry|
|
|
280
332
|
if entry.key?('error')
|
|
@@ -309,7 +361,7 @@ module Aspera
|
|
|
309
361
|
# @param path [String] file or folder path (end with "/" is like setting process_last_link)
|
|
310
362
|
# @param process_last_link [Boolean] if true, follow the last link
|
|
311
363
|
# @return [Hash] Result data
|
|
312
|
-
# @option return [
|
|
364
|
+
# @option return [Rest] :api REST client instance
|
|
313
365
|
# @option return [String] :file_id File identifier
|
|
314
366
|
def resolve_api_fid(top_file_id, path, process_last_link = false)
|
|
315
367
|
Aspera.assert_type(top_file_id, String)
|
|
@@ -445,7 +497,7 @@ module Aspera
|
|
|
445
497
|
# Add application specific tags (AoC)
|
|
446
498
|
@app_info[:api].add_ts_tags(transfer_spec: transfer_spec, app_info: @app_info) unless @app_info.nil?
|
|
447
499
|
# Add remote host info
|
|
448
|
-
if self.class.
|
|
500
|
+
if self.class.api_options[:standard_ports]
|
|
449
501
|
# Get default TCP/UDP ports and transfer user
|
|
450
502
|
transfer_spec.merge!(Transfer::Spec::AK_TSPEC_BASE)
|
|
451
503
|
# By default: same address as node API
|
data/lib/aspera/cli/formatter.rb
CHANGED
|
@@ -128,20 +128,34 @@ module Aspera
|
|
|
128
128
|
@spinner = nil
|
|
129
129
|
end
|
|
130
130
|
|
|
131
|
-
|
|
132
|
-
def long_operation_running(title = '')
|
|
131
|
+
def long_operation(title = nil, action: :spin)
|
|
133
132
|
return unless Environment.terminal?
|
|
134
|
-
if
|
|
133
|
+
return if %i[error data].include?(@options[:display])
|
|
134
|
+
|
|
135
|
+
# Handle the "delayed start" state
|
|
136
|
+
return @spinner = :starting if action == :spin && @spinner.nil?
|
|
137
|
+
|
|
138
|
+
# Cleanup if we try to stop a spinner that never actually started
|
|
139
|
+
@spinner = nil if action != :spin && @spinner == :starting
|
|
140
|
+
return if @spinner.nil?
|
|
141
|
+
|
|
142
|
+
# Initialize the real TTY object if it's currently just the :starting symbol
|
|
143
|
+
if @spinner == :starting
|
|
135
144
|
@spinner = TTY::Spinner.new('[:spinner] :title', format: :classic)
|
|
145
|
+
@spinner.update(title: '')
|
|
136
146
|
@spinner.start
|
|
137
147
|
end
|
|
138
|
-
@spinner.update(title: title)
|
|
139
|
-
@spinner.spin
|
|
140
|
-
end
|
|
141
148
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
149
|
+
@spinner.update(title: title) if title
|
|
150
|
+
|
|
151
|
+
case action
|
|
152
|
+
when :spin
|
|
153
|
+
@spinner.spin
|
|
154
|
+
when :success, :fail
|
|
155
|
+
action == :success ? @spinner.success : @spinner.error
|
|
156
|
+
@spinner.stop
|
|
157
|
+
@spinner = nil
|
|
158
|
+
end
|
|
145
159
|
end
|
|
146
160
|
|
|
147
161
|
def declare_options(options)
|
|
@@ -150,9 +164,9 @@ module Aspera
|
|
|
150
164
|
else
|
|
151
165
|
{}
|
|
152
166
|
end
|
|
167
|
+
options.declare(:display, 'Output only some information', allowed: DISPLAY_LEVELS, handler: {o: self, m: :option_handler}, default: :data)
|
|
153
168
|
options.declare(:format, 'Output format', allowed: DISPLAY_FORMATS, handler: {o: self, m: :option_handler}, default: :table)
|
|
154
169
|
options.declare(:output, 'Destination for results', handler: {o: self, m: :option_handler})
|
|
155
|
-
options.declare(:display, 'Output only some information', allowed: DISPLAY_LEVELS, handler: {o: self, m: :option_handler}, default: :info)
|
|
156
170
|
options.declare(
|
|
157
171
|
:fields, "Comma separated list of: fields, or #{SpecialValues::ALL}, or #{SpecialValues::DEF}", handler: {o: self, m: :option_handler},
|
|
158
172
|
allowed: [String, Array, Regexp, Proc],
|
|
@@ -178,6 +192,8 @@ module Aspera
|
|
|
178
192
|
@options[option_symbol] = value
|
|
179
193
|
# special handling of some options
|
|
180
194
|
case option_symbol
|
|
195
|
+
when :format
|
|
196
|
+
@options[:display] = value.eql?(:table) ? :info : :data
|
|
181
197
|
when :output
|
|
182
198
|
$stdout = if value.eql?('-')
|
|
183
199
|
STDOUT # rubocop:disable Style/GlobalStdStream
|
|
@@ -197,7 +213,7 @@ module Aspera
|
|
|
197
213
|
nil
|
|
198
214
|
end
|
|
199
215
|
|
|
200
|
-
#
|
|
216
|
+
# Main output method
|
|
201
217
|
# data: for requested data, not displayed if level==error
|
|
202
218
|
# info: additional info, displayed if level==info
|
|
203
219
|
# error: always displayed on stderr
|
data/lib/aspera/cli/manager.rb
CHANGED
|
@@ -330,7 +330,7 @@ module Aspera
|
|
|
330
330
|
|
|
331
331
|
# @param descr [String] description for help
|
|
332
332
|
# @param mandatory [Boolean] if true, raise error if option not set
|
|
333
|
-
# @param multiple [Boolean] if true, return remaining arguments (Array)
|
|
333
|
+
# @param multiple [Boolean] if true, return remaining arguments (Array) unil END
|
|
334
334
|
# @param accept_list [Array, NilClass] list of allowed values (Symbol)
|
|
335
335
|
# @param validation [Class, Array, NilClass] Accepted value type(s) or list of Symbols
|
|
336
336
|
# @param aliases [Hash] map of aliases: key = alias, value = real value
|
|
@@ -345,10 +345,19 @@ module Aspera
|
|
|
345
345
|
descr = "#{descr} (#{validation.join(', ')})" unless validation.nil? || validation.eql?(Allowed::TYPES_STRING)
|
|
346
346
|
result =
|
|
347
347
|
if !@unprocessed_cmd_line_arguments.empty?
|
|
348
|
-
|
|
349
|
-
|
|
348
|
+
if multiple
|
|
349
|
+
index = @unprocessed_cmd_line_arguments.index(SpecialValues::EOA)
|
|
350
|
+
if index.nil?
|
|
351
|
+
values = @unprocessed_cmd_line_arguments.shift(@unprocessed_cmd_line_arguments.length)
|
|
352
|
+
else
|
|
353
|
+
values = @unprocessed_cmd_line_arguments.shift(index)
|
|
354
|
+
@unprocessed_cmd_line_arguments.shift # remove EOA
|
|
355
|
+
end
|
|
356
|
+
else
|
|
357
|
+
values = [@unprocessed_cmd_line_arguments.shift]
|
|
358
|
+
end
|
|
350
359
|
values = values.map{ |v| ExtendedValue.instance.evaluate(v, context: "argument: #{descr}", allowed: validation)}
|
|
351
|
-
#
|
|
360
|
+
# If expecting list and only one arg of type array : it is the list
|
|
352
361
|
values = values.first if multiple && values.length.eql?(1) && values.first.is_a?(Array)
|
|
353
362
|
if accept_list
|
|
354
363
|
allowed_values = [].concat(accept_list)
|
|
@@ -233,8 +233,7 @@ module Aspera
|
|
|
233
233
|
defaults: {workspace: nil},
|
|
234
234
|
scope: @scope,
|
|
235
235
|
subpath: aoc_base_path,
|
|
236
|
-
secret_finder: config
|
|
237
|
-
progress_disp: formatter
|
|
236
|
+
secret_finder: config
|
|
238
237
|
))
|
|
239
238
|
end
|
|
240
239
|
|
|
@@ -256,7 +255,10 @@ module Aspera
|
|
|
256
255
|
# @param hash [Hash,nil] Optional base `Hash` (modified)
|
|
257
256
|
# @param string [Boolean] `true` to set key as `String`, else as `Symbol`
|
|
258
257
|
# @param name [Boolean] Include name
|
|
259
|
-
# @return [Hash
|
|
258
|
+
# @return [Hash{Symbol, String => String}] the modified hash containing:
|
|
259
|
+
# * `workspace_id` [String] the unique identifier.
|
|
260
|
+
# * `workspace_name` [String] (optional) the name, included if +name+ is true.
|
|
261
|
+
# @note The key type (String or Symbol) depends on the +string+ parameter.
|
|
260
262
|
def workspace_id_hash(hash = nil, string: false, name: false)
|
|
261
263
|
info = aoc_api.workspace
|
|
262
264
|
hash = {} if hash.nil?
|
|
@@ -300,9 +300,9 @@ module Aspera
|
|
|
300
300
|
remain_pages -= 1 unless remain_pages.nil?
|
|
301
301
|
break if remain_pages == 0
|
|
302
302
|
offset += page_result[items_key].length
|
|
303
|
-
|
|
303
|
+
RestParameters.instance.spinner_cb.call("#{result.length} / #{total_count || '?'}")
|
|
304
304
|
end
|
|
305
|
-
|
|
305
|
+
RestParameters.instance.spinner_cb.call(action: :success)
|
|
306
306
|
return result, total_count
|
|
307
307
|
end
|
|
308
308
|
|
|
@@ -191,7 +191,8 @@ module Aspera
|
|
|
191
191
|
def setup_rest_and_transfer_runtime
|
|
192
192
|
RestParameters.instance.user_agent = Info::CMD_NAME
|
|
193
193
|
RestParameters.instance.progress_bar = @progress_bar
|
|
194
|
-
RestParameters.instance.session_cb =
|
|
194
|
+
RestParameters.instance.session_cb = ->(http_session){update_http_session(http_session)}
|
|
195
|
+
RestParameters.instance.spinner_cb = ->(title = nil, action: :spin){formatter.long_operation(title, action: action)}
|
|
195
196
|
# Check http options that are global
|
|
196
197
|
keys_to_delete = []
|
|
197
198
|
@option_http_options.each do |k, v|
|
|
@@ -164,10 +164,10 @@ module Aspera
|
|
|
164
164
|
loop do
|
|
165
165
|
result = @api_v5.read("jobs/#{job_id}", {type: :formatted})
|
|
166
166
|
break unless Api::Faspex::JOB_RUNNING.include?(result['status'])
|
|
167
|
-
|
|
167
|
+
RestParameters.instance.spinner_cb.call(result['status'])
|
|
168
168
|
sleep(0.5)
|
|
169
169
|
end
|
|
170
|
-
|
|
170
|
+
RestParameters.instance.spinner_cb.call(action: :success)
|
|
171
171
|
return result
|
|
172
172
|
end
|
|
173
173
|
|
|
@@ -192,6 +192,16 @@ module Aspera
|
|
|
192
192
|
return list.select(&filter), total
|
|
193
193
|
end
|
|
194
194
|
|
|
195
|
+
# Build query to get package recipients based on package info in case of shared inbox or workgroup recipient
|
|
196
|
+
# @param package_id [String] the package id to get info from
|
|
197
|
+
def recipient_query(package_id)
|
|
198
|
+
package_info = @api_v5.read("packages/#{package_id}")
|
|
199
|
+
base_query = {}
|
|
200
|
+
base_query['recipient_workgroup_id'] = package_info['recipients'].first['id'] if WORKGROUP_TYPES.include?(package_info['recipients'].first['recipient_type'])
|
|
201
|
+
base_query['recipient_user_id'] = package_info['recipients'].first['id'] if package_info['recipients'].first['recipient_type'].eql?('user')
|
|
202
|
+
base_query
|
|
203
|
+
end
|
|
204
|
+
|
|
195
205
|
def package_receive(package_ids)
|
|
196
206
|
# prepare persistency if needed
|
|
197
207
|
skip_ids_persistency = nil
|
|
@@ -241,7 +251,7 @@ module Aspera
|
|
|
241
251
|
type: Api::Faspex.box_type(box),
|
|
242
252
|
transfer_type: Api::Faspex::TRANSFER_CONNECT
|
|
243
253
|
}
|
|
244
|
-
download_params[:recipient_workgroup_id] = lookup_entity_by_field(api: @api_v5, entity: options.get_option(:group_type), value: box)['id'] if !Api::Faspex::API_LIST_MAILBOX_TYPES.include?(box) && box != SpecialValues::ALL
|
|
254
|
+
# download_params[:recipient_workgroup_id] = lookup_entity_by_field(api: @api_v5, entity: options.get_option(:group_type), value: box)['id'] if !Api::Faspex::API_LIST_MAILBOX_TYPES.include?(box) && box != SpecialValues::ALL
|
|
245
255
|
packages.each do |package|
|
|
246
256
|
pkg_id = package['id']
|
|
247
257
|
formatter.display_status("Receiving package #{pkg_id}")
|
|
@@ -249,7 +259,7 @@ module Aspera
|
|
|
249
259
|
transfer_spec = @api_v5.call(
|
|
250
260
|
operation: 'POST',
|
|
251
261
|
subpath: "packages/#{pkg_id}/transfer_spec/download",
|
|
252
|
-
query: download_params,
|
|
262
|
+
query: download_params.merge(recipient_query(pkg_id)),
|
|
253
263
|
content_type: Mime::JSON,
|
|
254
264
|
body: param_file_list,
|
|
255
265
|
headers: {'Accept' => Mime::JSON}
|
|
@@ -319,9 +329,9 @@ module Aspera
|
|
|
319
329
|
|
|
320
330
|
# Browse a folder
|
|
321
331
|
# @param browse_endpoint [String] the endpoint to browse
|
|
322
|
-
def browse_folder(browse_endpoint)
|
|
332
|
+
def browse_folder(browse_endpoint, base_query = {})
|
|
323
333
|
folders_to_process = [options.get_next_argument('folder path', default: '/')]
|
|
324
|
-
query = query_read_delete(default: {})
|
|
334
|
+
query = base_query.merge(query_read_delete(default: {}))
|
|
325
335
|
filters = query.delete('filters'){{}}
|
|
326
336
|
Aspera.assert_type(filters, Hash)
|
|
327
337
|
filters['basenames'] ||= []
|
|
@@ -357,7 +367,7 @@ module Aspera
|
|
|
357
367
|
end
|
|
358
368
|
folders_to_process.concat(data['items'].select{ |i| i['type'].eql?('directory')}.map{ |i| i['path']}) if recursive
|
|
359
369
|
if use_paging
|
|
360
|
-
iteration_token = http[Api::Faspex::
|
|
370
|
+
iteration_token = http[Api::Faspex::HEADER_X_NEXT_ITER_TOKEN]
|
|
361
371
|
break if iteration_token.nil? || iteration_token.empty?
|
|
362
372
|
query['iteration_token'] = iteration_token
|
|
363
373
|
else
|
|
@@ -365,12 +375,11 @@ module Aspera
|
|
|
365
375
|
break if data['item_count'].eql?(0)
|
|
366
376
|
query['offset'] += data['item_count']
|
|
367
377
|
end
|
|
368
|
-
|
|
378
|
+
RestParameters.instance.spinner_cb.call(all_items.count)
|
|
369
379
|
end
|
|
370
380
|
query.delete('iteration_token')
|
|
371
381
|
end
|
|
372
|
-
|
|
373
|
-
|
|
382
|
+
RestParameters.instance.spinner_cb.call(action: :success)
|
|
374
383
|
return Main.result_object_list(all_items, total: total_count)
|
|
375
384
|
end
|
|
376
385
|
|
|
@@ -384,7 +393,7 @@ module Aspera
|
|
|
384
393
|
when :show
|
|
385
394
|
return Main.result_single_object(@api_v5.read("packages/#{package_id}"))
|
|
386
395
|
when :browse
|
|
387
|
-
return browse_folder("packages/#{package_id}/files/#{Api::Faspex.box_type(options.get_option(:box))}")
|
|
396
|
+
return browse_folder("packages/#{package_id}/files/#{Api::Faspex.box_type(options.get_option(:box))}", recipient_query(package_id))
|
|
388
397
|
when :status
|
|
389
398
|
status_list = options.get_next_argument('list of states, or nothing', mandatory: false, validation: Array)
|
|
390
399
|
status = wait_package_status(package_id, status_list: status_list)
|
|
@@ -447,6 +456,8 @@ module Aspera
|
|
|
447
456
|
res_id_query = {'all': true}
|
|
448
457
|
when :nodes
|
|
449
458
|
available_commands += %i[shared_folders browse]
|
|
459
|
+
when :jobs
|
|
460
|
+
exec_args[:display_fields] = %w[id job_name job_type status]
|
|
450
461
|
end
|
|
451
462
|
res_command = options.get_next_command(available_commands)
|
|
452
463
|
return Main.result_value_list(Api::Faspex::EMAIL_NOTIF_LIST, name: 'email_id') if res_command.eql?(:list) && res_sym.eql?(:email_notifications)
|
|
@@ -719,7 +730,8 @@ module Aspera
|
|
|
719
730
|
end
|
|
720
731
|
SHARED_INBOX_MEMBER_LEVELS = %i[submit_only standard shared_inbox_admin].freeze
|
|
721
732
|
ACCOUNT_TYPES = %w{local_user saml_user self_registered_user external_user}.freeze
|
|
722
|
-
|
|
733
|
+
WORKGROUP_TYPES = %w{workgroup shared_inbox}.freeze
|
|
734
|
+
CONTACT_TYPES = (WORKGROUP_TYPES + %w{distribution_list user external_user}).freeze
|
|
723
735
|
private_constant :SHARED_INBOX_MEMBER_LEVELS, :ACCOUNT_TYPES, :CONTACT_TYPES
|
|
724
736
|
end
|
|
725
737
|
end
|
|
@@ -93,12 +93,9 @@ module Aspera
|
|
|
93
93
|
options.declare(:validator, 'Identifier of validator (optional for central)')
|
|
94
94
|
options.declare(:asperabrowserurl, 'URL for simple aspera web ui', default: 'https://asperabrowser.mybluemix.net')
|
|
95
95
|
options.declare(
|
|
96
|
-
:
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
options.declare(
|
|
100
|
-
:node_cache, 'Gen4: Set to no to force actual file system read', allowed: Allowed::TYPES_BOOLEAN,
|
|
101
|
-
handler: {o: Api::Node, m: :use_node_cache}
|
|
96
|
+
:node_api, 'Gen4: standard_ports: Use standard FASP ports (true) or get from node API (false). cache: Set to false to force actual file system read',
|
|
97
|
+
allowed: Hash,
|
|
98
|
+
handler: {o: Api::Node, m: :api_options}
|
|
102
99
|
)
|
|
103
100
|
options.declare(:root_id, 'Gen4: File id of top folder when using access key (override AK root id)')
|
|
104
101
|
options.declare(:dynamic_key, 'Private key PEM to use for dynamic key auth', handler: {o: Api::Node, m: :use_dynamic_key})
|
|
@@ -242,14 +239,14 @@ module Aspera
|
|
|
242
239
|
break if all_items.count >= total_count
|
|
243
240
|
offset += items.count
|
|
244
241
|
query['skip'] = offset
|
|
245
|
-
|
|
242
|
+
RestParameters.instance.spinner_cb.call(all_items.count)
|
|
246
243
|
end
|
|
247
244
|
query.delete('skip')
|
|
248
245
|
end
|
|
249
246
|
@prefixer&.remove_in_object_list!(all_items)
|
|
250
247
|
return Main.result_object_list(all_items)
|
|
251
248
|
ensure
|
|
252
|
-
|
|
249
|
+
RestParameters.instance.spinner_cb.call(action: :success)
|
|
253
250
|
end
|
|
254
251
|
|
|
255
252
|
# Create async transfer spec request from direction and folders
|
|
@@ -388,7 +385,7 @@ module Aspera
|
|
|
388
385
|
when :transport
|
|
389
386
|
return Main.result_single_object(@api_node.transport_params)
|
|
390
387
|
when :spec
|
|
391
|
-
return Main.result_single_object(@api_node.base_spec)
|
|
388
|
+
return Main.result_single_object(@api_node.base_spec, fields: Formatter.all_but(Transfer::Spec::SPECIFIC))
|
|
392
389
|
end
|
|
393
390
|
Aspera.error_unreachable_line
|
|
394
391
|
end
|
|
@@ -523,7 +520,7 @@ module Aspera
|
|
|
523
520
|
return Main.result_text(result[:password])
|
|
524
521
|
when :browse
|
|
525
522
|
apifid = apifid_from_next_arg(top_file_id)
|
|
526
|
-
file_info = apifid[:api].read("files/#{apifid[:file_id]}",
|
|
523
|
+
file_info = apifid[:api].read("files/#{apifid[:file_id]}", headers: Api::Node.add_cache_control)
|
|
527
524
|
unless file_info['type'].eql?('folder')
|
|
528
525
|
# a single file
|
|
529
526
|
return Main.result_object_list([file_info], fields: GEN4_LS_FIELDS)
|
data/lib/aspera/cli/version.rb
CHANGED
data/lib/aspera/cli/wizard.rb
CHANGED
|
@@ -62,7 +62,7 @@ module Aspera
|
|
|
62
62
|
detection_info = nil
|
|
63
63
|
begin
|
|
64
64
|
Log.log.debug{"detecting #{plugin_name_sym} at #{app_url}"}
|
|
65
|
-
|
|
65
|
+
RestParameters.instance.spinner_cb.call(plugin_name_sym.to_s)
|
|
66
66
|
detection_info = plugin_klass.detect(app_url)
|
|
67
67
|
rescue OpenSSL::SSL::SSLError => e
|
|
68
68
|
Log.log.warn(e.message)
|
|
@@ -78,6 +78,7 @@ module Aspera
|
|
|
78
78
|
# If there is a redirect, then the detector can override the url.
|
|
79
79
|
found_apps.push({product: plugin_name_sym, name: app_name, url: app_url, version: 'unknown'}.merge(detection_info))
|
|
80
80
|
end
|
|
81
|
+
RestParameters.instance.spinner_cb.call(action: :success)
|
|
81
82
|
raise "No known application found at #{app_url}" if found_apps.empty?
|
|
82
83
|
Aspera.assert(found_apps.all?{ |a| a.keys.all?(Symbol)})
|
|
83
84
|
return found_apps
|
data/lib/aspera/rest.rb
CHANGED
|
@@ -33,7 +33,7 @@ module Aspera
|
|
|
33
33
|
class RestParameters
|
|
34
34
|
include Singleton
|
|
35
35
|
|
|
36
|
-
attr_accessor :user_agent, :download_partial_suffix, :retry_on_error, :retry_on_timeout, :retry_on_unavailable, :retry_max, :retry_sleep, :session_cb, :progress_bar
|
|
36
|
+
attr_accessor :user_agent, :download_partial_suffix, :retry_on_error, :retry_on_timeout, :retry_on_unavailable, :retry_max, :retry_sleep, :session_cb, :progress_bar, :spinner_cb
|
|
37
37
|
|
|
38
38
|
private
|
|
39
39
|
|
|
@@ -47,6 +47,7 @@ module Aspera
|
|
|
47
47
|
@retry_sleep = 4
|
|
48
48
|
@session_cb = nil
|
|
49
49
|
@progress_bar = nil
|
|
50
|
+
@spinner_cb = nil
|
|
50
51
|
end
|
|
51
52
|
end
|
|
52
53
|
|
|
@@ -88,8 +89,9 @@ module Aspera
|
|
|
88
89
|
query
|
|
89
90
|
end
|
|
90
91
|
|
|
91
|
-
# Build URI from URL and parameters and check it is http or https
|
|
92
|
-
# Check
|
|
92
|
+
# Build URI from URL and parameters and check it is `http` or `https`.
|
|
93
|
+
# Check if php style is specified.
|
|
94
|
+
# `nil` values in query result in key without value, e.g. `?a`, while empty string values result in `?a=`.
|
|
93
95
|
# @param url [String] The URL without query.
|
|
94
96
|
# @param query [Hash,Array,String] The query.
|
|
95
97
|
def build_uri(url, query)
|
|
@@ -105,18 +107,19 @@ module Aspera
|
|
|
105
107
|
URI.encode_www_form(h_to_query_array(query))
|
|
106
108
|
when Array
|
|
107
109
|
Aspera.assert(query.all?{ |i| i.is_a?(Array) && i.length.eql?(2)}){'Query must be array of arrays of 2 elements'}
|
|
108
|
-
URI.encode_www_form(query)
|
|
110
|
+
URI.encode_www_form(query) # remove nil values
|
|
109
111
|
else Aspera.error_unexpected_value(query.class){'query type'}
|
|
110
112
|
end.gsub('%5B%5D=', '[]=')
|
|
111
113
|
# [] is allowed in url parameters
|
|
112
114
|
uri
|
|
113
115
|
end
|
|
114
116
|
|
|
117
|
+
# Support array for query parameter, there is no standard.
|
|
118
|
+
# Either p=1&p=2 (default)
|
|
119
|
+
# or p[]=1&p[]=2 (if `:x_array_php_style` is set to true in query)
|
|
115
120
|
# @param query [Hash] HTTP query as hash
|
|
116
121
|
def h_to_query_array(query)
|
|
117
122
|
Aspera.assert_type(query, Hash)
|
|
118
|
-
# Support array for query parameter, there is no standard.
|
|
119
|
-
# Either p[]=1&p[]=2, or p=1&p=2
|
|
120
123
|
suffix = query.delete(:x_array_php_style) ? '[]' : nil
|
|
121
124
|
query.each_with_object([]) do |(k, v), query_array|
|
|
122
125
|
case v
|
|
@@ -412,7 +415,7 @@ module Aspera
|
|
|
412
415
|
http_session.request(req) do |response|
|
|
413
416
|
result_http = response
|
|
414
417
|
result_mime = self.class.parse_header(result_http['Content-Type'] || Mime::TEXT)[:type]
|
|
415
|
-
Log.log.debug{"response: code=#{result_http.code}, mime=#{result_mime},
|
|
418
|
+
Log.log.debug{"response: code=#{result_http.code}, mime=#{result_mime}, content-type=#{response['Content-Type']}"}
|
|
416
419
|
# JSON data needs to be parsed, in case it contains an error code
|
|
417
420
|
if !save_to_file.nil? &&
|
|
418
421
|
result_http.code.to_s.start_with?('2') &&
|
|
@@ -453,12 +456,10 @@ module Aspera
|
|
|
453
456
|
# TODO : related to mime type encoding ?
|
|
454
457
|
# result_http.body.force_encoding('UTF-8') if result_http.body.is_a?(String)
|
|
455
458
|
# Log.log.debug{"result: body=#{result_http.body}"}
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
result_data = result_http.body
|
|
461
|
-
end
|
|
459
|
+
result_data = result_http.body
|
|
460
|
+
Log.dump(:result_data_raw, result_data, level: :trace1)
|
|
461
|
+
result_data = JSON.parse(result_data) if Mime.json?(result_mime) && !result_data.nil? && !result_data.empty?
|
|
462
|
+
Log.dump(:result_data, result_data)
|
|
462
463
|
RestErrorAnalyzer.instance.raise_on_error(req, result_data, result_http)
|
|
463
464
|
unless file_saved || save_to_file.nil?
|
|
464
465
|
FileUtils.mkdir_p(File.dirname(save_to_file))
|
|
@@ -536,27 +537,39 @@ module Aspera
|
|
|
536
537
|
|
|
537
538
|
# Create: `POST`
|
|
538
539
|
def create(subpath, params, **kwargs)
|
|
539
|
-
|
|
540
|
+
kwargs[:headers] ||= {}
|
|
541
|
+
kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept')
|
|
542
|
+
kwargs[:content_type] = Mime::JSON unless kwargs.key?(:content_type)
|
|
543
|
+
return call(operation: 'POST', subpath: subpath, body: params, **kwargs)
|
|
540
544
|
end
|
|
541
545
|
|
|
542
546
|
# Read: `GET`
|
|
543
547
|
def read(subpath, query = nil, **kwargs)
|
|
544
|
-
|
|
548
|
+
kwargs[:headers] ||= {}
|
|
549
|
+
kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept')
|
|
550
|
+
return call(operation: 'GET', subpath: subpath, query: query, **kwargs)
|
|
545
551
|
end
|
|
546
552
|
|
|
547
553
|
# Update: `PUT`
|
|
548
554
|
def update(subpath, params, **kwargs)
|
|
549
|
-
|
|
555
|
+
kwargs[:headers] ||= {}
|
|
556
|
+
kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept')
|
|
557
|
+
kwargs[:content_type] = Mime::JSON unless kwargs.key?(:content_type)
|
|
558
|
+
return call(operation: 'PUT', subpath: subpath, body: params, **kwargs)
|
|
550
559
|
end
|
|
551
560
|
|
|
552
561
|
# Delete: `DELETE`
|
|
553
562
|
def delete(subpath, params = nil, **kwargs)
|
|
554
|
-
|
|
563
|
+
kwargs[:headers] ||= {}
|
|
564
|
+
kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept')
|
|
565
|
+
return call(operation: 'DELETE', subpath: subpath, query: params, **kwargs)
|
|
555
566
|
end
|
|
556
567
|
|
|
557
568
|
# Cancel: `CANCEL`
|
|
558
569
|
def cancel(subpath, **kwargs)
|
|
559
|
-
|
|
570
|
+
kwargs[:headers] ||= {}
|
|
571
|
+
kwargs[:headers]['Accept'] = Mime::JSON unless kwargs[:headers].key?('Accept')
|
|
572
|
+
return call(operation: 'CANCEL', subpath: subpath, **kwargs)
|
|
560
573
|
end
|
|
561
574
|
|
|
562
575
|
# Query entity by general search (read with parameter `q`)
|
data/lib/aspera/transfer/spec.rb
CHANGED
|
@@ -25,6 +25,7 @@ module Aspera
|
|
|
25
25
|
TRANSPORT_FIELDS = (%w[remote_host] + AK_TSPEC_BASE.keys + WSS_FIELDS).freeze
|
|
26
26
|
# reserved tag for Aspera
|
|
27
27
|
TAG_RESERVED = 'aspera'
|
|
28
|
+
SPECIFIC = %w{token paths direction source_root destination_root}.freeze
|
|
28
29
|
class << self
|
|
29
30
|
# wrong def in transferd
|
|
30
31
|
POLICY_FIX = {
|
data.tar.gz.sig
CHANGED
|
Binary file
|
metadata
CHANGED
metadata.gz.sig
CHANGED
|
Binary file
|