aspera-cli 4.14.0 → 4.16.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +29 -3
- data/CHANGELOG.md +300 -185
- data/CONTRIBUTING.md +74 -23
- data/README.md +2346 -1619
- data/bin/ascli +16 -25
- data/bin/asession +15 -15
- data/examples/dascli +2 -2
- data/examples/proxy.pac +1 -1
- data/lib/aspera/aoc.rb +216 -150
- data/lib/aspera/ascmd.rb +25 -18
- data/lib/aspera/assert.rb +45 -0
- data/lib/aspera/cli/basic_auth_plugin.rb +9 -6
- data/lib/aspera/cli/error.rb +17 -0
- data/lib/aspera/cli/extended_value.rb +51 -16
- data/lib/aspera/cli/formatter.rb +276 -174
- data/lib/aspera/cli/hints.rb +81 -0
- data/lib/aspera/cli/main.rb +114 -147
- data/lib/aspera/cli/manager.rb +181 -136
- data/lib/aspera/cli/plugin.rb +82 -64
- data/lib/aspera/cli/plugins/alee.rb +0 -1
- data/lib/aspera/cli/plugins/aoc.rb +327 -331
- data/lib/aspera/cli/plugins/ats.rb +12 -8
- data/lib/aspera/cli/plugins/bss.rb +2 -2
- data/lib/aspera/cli/plugins/config.rb +575 -439
- data/lib/aspera/cli/plugins/console.rb +40 -0
- data/lib/aspera/cli/plugins/cos.rb +4 -5
- data/lib/aspera/cli/plugins/faspex.rb +111 -92
- data/lib/aspera/cli/plugins/faspex5.rb +245 -182
- data/lib/aspera/cli/plugins/node.rb +239 -160
- data/lib/aspera/cli/plugins/orchestrator.rb +56 -19
- data/lib/aspera/cli/plugins/preview.rb +54 -38
- data/lib/aspera/cli/plugins/server.rb +63 -20
- data/lib/aspera/cli/plugins/shares.rb +64 -38
- data/lib/aspera/cli/sync_actions.rb +68 -0
- data/lib/aspera/cli/transfer_agent.rb +64 -67
- data/lib/aspera/cli/transfer_progress.rb +73 -0
- data/lib/aspera/cli/version.rb +1 -1
- data/lib/aspera/colors.rb +3 -1
- data/lib/aspera/command_line_builder.rb +27 -22
- data/lib/aspera/cos_node.rb +6 -4
- data/lib/aspera/coverage.rb +22 -0
- data/lib/aspera/data_repository.rb +33 -2
- data/lib/aspera/environment.rb +21 -8
- data/lib/aspera/fasp/agent_alpha.rb +116 -0
- data/lib/aspera/fasp/agent_base.rb +40 -76
- data/lib/aspera/fasp/agent_connect.rb +21 -22
- data/lib/aspera/fasp/agent_direct.rb +169 -179
- data/lib/aspera/fasp/agent_httpgw.rb +200 -195
- data/lib/aspera/fasp/agent_node.rb +43 -35
- data/lib/aspera/fasp/agent_trsdk.rb +124 -41
- data/lib/aspera/fasp/error_info.rb +2 -2
- data/lib/aspera/fasp/faux_file.rb +52 -0
- data/lib/aspera/fasp/installation.rb +89 -191
- data/lib/aspera/fasp/management.rb +249 -0
- data/lib/aspera/fasp/parameters.rb +86 -47
- data/lib/aspera/fasp/parameters.yaml +75 -8
- data/lib/aspera/fasp/products.rb +162 -0
- data/lib/aspera/fasp/resume_policy.rb +7 -5
- data/lib/aspera/fasp/sync.rb +273 -0
- data/lib/aspera/fasp/transfer_spec.rb +10 -8
- data/lib/aspera/fasp/uri.rb +6 -6
- data/lib/aspera/faspex_gw.rb +11 -8
- data/lib/aspera/faspex_postproc.rb +8 -7
- data/lib/aspera/hash_ext.rb +2 -2
- data/lib/aspera/id_generator.rb +3 -1
- data/lib/aspera/json_rpc.rb +51 -0
- data/lib/aspera/keychain/encrypted_hash.rb +46 -11
- data/lib/aspera/keychain/macos_security.rb +15 -13
- data/lib/aspera/line_logger.rb +23 -0
- data/lib/aspera/log.rb +61 -19
- data/lib/aspera/nagios.rb +7 -2
- data/lib/aspera/node.rb +105 -21
- data/lib/aspera/node_simulator.rb +214 -0
- data/lib/aspera/oauth.rb +57 -36
- data/lib/aspera/open_application.rb +4 -4
- data/lib/aspera/persistency_action_once.rb +13 -14
- data/lib/aspera/persistency_folder.rb +5 -4
- data/lib/aspera/preview/file_types.rb +56 -268
- data/lib/aspera/preview/generator.rb +28 -39
- data/lib/aspera/preview/options.rb +2 -0
- data/lib/aspera/preview/terminal.rb +36 -16
- data/lib/aspera/preview/utils.rb +23 -29
- data/lib/aspera/proxy_auto_config.rb +6 -3
- data/lib/aspera/rest.rb +127 -80
- data/lib/aspera/rest_call_error.rb +1 -1
- data/lib/aspera/rest_error_analyzer.rb +16 -14
- data/lib/aspera/rest_errors_aspera.rb +39 -34
- data/lib/aspera/secret_hider.rb +18 -17
- data/lib/aspera/ssh.rb +10 -5
- data/lib/aspera/temp_file_manager.rb +11 -4
- data/lib/aspera/web_auth.rb +10 -7
- data/lib/aspera/web_server_simple.rb +11 -5
- data.tar.gz.sig +0 -0
- metadata +108 -39
- metadata.gz.sig +0 -0
- data/lib/aspera/cli/listener/line_dump.rb +0 -19
- data/lib/aspera/cli/listener/logger.rb +0 -22
- data/lib/aspera/cli/listener/progress.rb +0 -50
- data/lib/aspera/cli/listener/progress_multi.rb +0 -84
- data/lib/aspera/cli/plugins/sync.rb +0 -44
- data/lib/aspera/fasp/listener.rb +0 -13
- data/lib/aspera/sync.rb +0 -213
data/CONTRIBUTING.md
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
# Contributing
|
2
|
-
<!-- cspell:words passin
|
2
|
+
<!-- cspell:words passin -->
|
3
3
|
## Reporting Issues and Vulnerabilities
|
4
4
|
|
5
5
|
You can report issues at <https://github.com/IBM/aspera-cli/issues>
|
@@ -32,7 +32,7 @@ If you want to contribute, please:
|
|
32
32
|
- Add tests for it. This is important so I don't break it in a future version unintentionally.
|
33
33
|
- **Bonus Points** go out to anyone who also updates `CHANGELOG.md` :)
|
34
34
|
- Send a pull request on GitHub.
|
35
|
-
- run
|
35
|
+
- run `rubocop` to comply for coding standards
|
36
36
|
|
37
37
|
## Architecture
|
38
38
|
|
@@ -42,32 +42,81 @@ Architecture:
|
|
42
42
|
|
43
43
|
![Architecture](docs/architecture.png)
|
44
44
|
|
45
|
-
The entry point is lib/aspera/cli/main.rb
|
45
|
+
The entry point is: `lib/aspera/cli/main.rb`.
|
46
46
|
|
47
|
-
Plugins are located in lib/aspera/cli/plugins
|
47
|
+
Plugins are located in: `lib/aspera/cli/plugins`.
|
48
48
|
|
49
|
-
Transfer agents: lib/aspera/fasp
|
49
|
+
Transfer agents, in: `lib/aspera/fasp`.
|
50
|
+
|
51
|
+
## Ruby version
|
52
|
+
|
53
|
+
Install ruby with [RVM](https://rvm.io).
|
54
|
+
|
55
|
+
Cleanup installed gems in `rvm` to start fresh:
|
56
|
+
|
57
|
+
```bash
|
58
|
+
make cleanup_gems
|
59
|
+
```
|
60
|
+
|
61
|
+
## Tool chain
|
62
|
+
|
63
|
+
TODO: document installation of tool chain.
|
64
|
+
|
65
|
+
### Documentation
|
66
|
+
|
67
|
+
Documentation is generated with `pandoc` and `LaTeX`.
|
68
|
+
IBM font `Plex` is used, see [IBM Plex](https://www.ibm.com/plex/).
|
50
69
|
|
51
70
|
## Running Tests
|
52
71
|
|
53
|
-
First, a testing
|
72
|
+
First, a testing configuration file must be created, from project top folder:
|
54
73
|
|
55
74
|
```bash
|
56
|
-
mkdir
|
57
|
-
cp docs/test_env.conf
|
75
|
+
mkdir ~/some_secure_folder
|
76
|
+
cp docs/test_env.conf ~/some_secure_folder/.
|
58
77
|
```
|
59
78
|
|
60
|
-
Fill
|
79
|
+
Fill `~/some_secure_folder/test_env.conf` with system URLs and credentials for tests.
|
61
80
|
|
62
|
-
|
81
|
+
Then tell where this file is located:
|
63
82
|
|
64
83
|
```bash
|
65
|
-
|
66
|
-
|
84
|
+
export ASPERA_CLI_TEST_CONF_FILE=~/some_secure_folder/test_env.conf
|
85
|
+
```
|
86
|
+
|
87
|
+
This project uses a `Makefile` for tests:
|
88
|
+
|
89
|
+
```bash
|
90
|
+
make test
|
67
91
|
```
|
68
92
|
|
69
93
|
When new commands are added to the CLI, new tests shall be added to the test suite.
|
70
94
|
|
95
|
+
### Special tests
|
96
|
+
|
97
|
+
Some gems are optional: `rmagick` and `grpc`, as they require compilation of native code which may cause problems.
|
98
|
+
By default, tests that use those gems are skipped.
|
99
|
+
To run them: `make optional`.
|
100
|
+
Those tests also require the optional gems to be installed: `make install_optional_gems`.
|
101
|
+
|
102
|
+
Some other tests require interactive input. To run them: `make interactive`
|
103
|
+
|
104
|
+
To run every test: `make full`
|
105
|
+
|
106
|
+
### Pre-release tests
|
107
|
+
|
108
|
+
For preparation of a release, do the following:
|
109
|
+
|
110
|
+
1. Select a ruby version to test with: `rvm use 3.2.2`
|
111
|
+
2. Remove all gems: `make clean_gems`
|
112
|
+
3. Starts tests: `make test`
|
113
|
+
4. Install optional gems: `make install_optional_gems`
|
114
|
+
5. Test remaining cases: `cd tests` and then `make optional` and `make interactive`
|
115
|
+
|
116
|
+
This ensures that optional gems do not prevent `ascli` to run.
|
117
|
+
|
118
|
+
To test additional Ruby version, repeat the procedure with other Ruby versions.
|
119
|
+
|
71
120
|
## Coverage
|
72
121
|
|
73
122
|
A coverage report can be generated in folder `coverage` using gem `SimpleCov`.
|
@@ -102,7 +151,7 @@ Then procedure is as follows:
|
|
102
151
|
gem cert --build maintainer@example.com
|
103
152
|
```
|
104
153
|
|
105
|
-
> **Note:**
|
154
|
+
> **Note:** The email must match the field `spec.email` in `aspera-cli.gemspec`
|
106
155
|
|
107
156
|
This creates two files in folder `/path/to/vault` (e.g. $HOME/.ssh):
|
108
157
|
|
@@ -111,7 +160,7 @@ Then procedure is as follows:
|
|
111
160
|
|
112
161
|
> **Note:** Alternatively, use an existing key or generate one, and then `make new-cert`
|
113
162
|
|
114
|
-
- The maintainer
|
163
|
+
- The maintainer builds the signed gem.
|
115
164
|
|
116
165
|
The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
|
117
166
|
|
@@ -187,26 +236,28 @@ A template is used as it allows some level of customization to tell where to tak
|
|
187
236
|
Then, to push to the image registry (both tags: version and `latest`):
|
188
237
|
|
189
238
|
```bash
|
190
|
-
make
|
239
|
+
make docker_push
|
191
240
|
```
|
192
241
|
|
193
242
|
### Specific version image build
|
194
243
|
|
195
|
-
|
244
|
+
To build a specific version: set envvar `GEM_VERSION` and `make` with option `-e`:
|
196
245
|
|
197
246
|
```bash
|
198
|
-
|
199
|
-
|
247
|
+
GEM_VERSION=4.11.0 make -e docker
|
248
|
+
GEM_VERSION=4.11.0 make -e docker_push
|
200
249
|
```
|
201
250
|
|
251
|
+
> **Note:** This does not use the gem locally, only the docker file is used locally, the gem is installed from rubygems.org.
|
252
|
+
|
202
253
|
### Development version image build
|
203
254
|
|
204
255
|
To build/push a beta/development container:
|
205
|
-
it does not create the `latest` tag, it uses the gem file generated locally.
|
256
|
+
it does not create the `latest` tag, it uses the gem file generated locally with a special version number.
|
206
257
|
|
207
258
|
```bash
|
208
|
-
make
|
209
|
-
make
|
259
|
+
make docker_beta
|
260
|
+
make docker_push_beta
|
210
261
|
```
|
211
262
|
|
212
263
|
## Single executable build
|
@@ -219,7 +270,7 @@ make single
|
|
219
270
|
|
220
271
|
This is based on tool `rubyc` (gem `ruby-packer`): either place the executable in folder `tmp`, or the architecture-specific version will be automatically downloaded.
|
221
272
|
|
222
|
-
> **Note:** Perl Findbin is needed, install on Linux with:
|
273
|
+
> **Note:** Perl `Findbin` is needed, install on Linux with:
|
223
274
|
|
224
275
|
```bash
|
225
276
|
sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
|
@@ -231,4 +282,4 @@ sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
|
|
231
282
|
- <https://github.com/rest-client/rest-client>
|
232
283
|
- <https://github.com/oauth-xx/oauth2>
|
233
284
|
- use gem Thor <http://whatisthor.com/> (or other standard Ruby CLI manager)
|
234
|
-
- Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (rubyc)
|
285
|
+
- Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (`rubyc`)
|