aspera-cli 4.14.0 → 4.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (104) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/BUGS.md +29 -3
  4. data/CHANGELOG.md +300 -185
  5. data/CONTRIBUTING.md +74 -23
  6. data/README.md +2346 -1619
  7. data/bin/ascli +16 -25
  8. data/bin/asession +15 -15
  9. data/examples/dascli +2 -2
  10. data/examples/proxy.pac +1 -1
  11. data/lib/aspera/aoc.rb +216 -150
  12. data/lib/aspera/ascmd.rb +25 -18
  13. data/lib/aspera/assert.rb +45 -0
  14. data/lib/aspera/cli/basic_auth_plugin.rb +9 -6
  15. data/lib/aspera/cli/error.rb +17 -0
  16. data/lib/aspera/cli/extended_value.rb +51 -16
  17. data/lib/aspera/cli/formatter.rb +276 -174
  18. data/lib/aspera/cli/hints.rb +81 -0
  19. data/lib/aspera/cli/main.rb +114 -147
  20. data/lib/aspera/cli/manager.rb +181 -136
  21. data/lib/aspera/cli/plugin.rb +82 -64
  22. data/lib/aspera/cli/plugins/alee.rb +0 -1
  23. data/lib/aspera/cli/plugins/aoc.rb +327 -331
  24. data/lib/aspera/cli/plugins/ats.rb +12 -8
  25. data/lib/aspera/cli/plugins/bss.rb +2 -2
  26. data/lib/aspera/cli/plugins/config.rb +575 -439
  27. data/lib/aspera/cli/plugins/console.rb +40 -0
  28. data/lib/aspera/cli/plugins/cos.rb +4 -5
  29. data/lib/aspera/cli/plugins/faspex.rb +111 -92
  30. data/lib/aspera/cli/plugins/faspex5.rb +245 -182
  31. data/lib/aspera/cli/plugins/node.rb +239 -160
  32. data/lib/aspera/cli/plugins/orchestrator.rb +56 -19
  33. data/lib/aspera/cli/plugins/preview.rb +54 -38
  34. data/lib/aspera/cli/plugins/server.rb +63 -20
  35. data/lib/aspera/cli/plugins/shares.rb +64 -38
  36. data/lib/aspera/cli/sync_actions.rb +68 -0
  37. data/lib/aspera/cli/transfer_agent.rb +64 -67
  38. data/lib/aspera/cli/transfer_progress.rb +73 -0
  39. data/lib/aspera/cli/version.rb +1 -1
  40. data/lib/aspera/colors.rb +3 -1
  41. data/lib/aspera/command_line_builder.rb +27 -22
  42. data/lib/aspera/cos_node.rb +6 -4
  43. data/lib/aspera/coverage.rb +22 -0
  44. data/lib/aspera/data_repository.rb +33 -2
  45. data/lib/aspera/environment.rb +21 -8
  46. data/lib/aspera/fasp/agent_alpha.rb +116 -0
  47. data/lib/aspera/fasp/agent_base.rb +40 -76
  48. data/lib/aspera/fasp/agent_connect.rb +21 -22
  49. data/lib/aspera/fasp/agent_direct.rb +169 -179
  50. data/lib/aspera/fasp/agent_httpgw.rb +200 -195
  51. data/lib/aspera/fasp/agent_node.rb +43 -35
  52. data/lib/aspera/fasp/agent_trsdk.rb +124 -41
  53. data/lib/aspera/fasp/error_info.rb +2 -2
  54. data/lib/aspera/fasp/faux_file.rb +52 -0
  55. data/lib/aspera/fasp/installation.rb +89 -191
  56. data/lib/aspera/fasp/management.rb +249 -0
  57. data/lib/aspera/fasp/parameters.rb +86 -47
  58. data/lib/aspera/fasp/parameters.yaml +75 -8
  59. data/lib/aspera/fasp/products.rb +162 -0
  60. data/lib/aspera/fasp/resume_policy.rb +7 -5
  61. data/lib/aspera/fasp/sync.rb +273 -0
  62. data/lib/aspera/fasp/transfer_spec.rb +10 -8
  63. data/lib/aspera/fasp/uri.rb +6 -6
  64. data/lib/aspera/faspex_gw.rb +11 -8
  65. data/lib/aspera/faspex_postproc.rb +8 -7
  66. data/lib/aspera/hash_ext.rb +2 -2
  67. data/lib/aspera/id_generator.rb +3 -1
  68. data/lib/aspera/json_rpc.rb +51 -0
  69. data/lib/aspera/keychain/encrypted_hash.rb +46 -11
  70. data/lib/aspera/keychain/macos_security.rb +15 -13
  71. data/lib/aspera/line_logger.rb +23 -0
  72. data/lib/aspera/log.rb +61 -19
  73. data/lib/aspera/nagios.rb +7 -2
  74. data/lib/aspera/node.rb +105 -21
  75. data/lib/aspera/node_simulator.rb +214 -0
  76. data/lib/aspera/oauth.rb +57 -36
  77. data/lib/aspera/open_application.rb +4 -4
  78. data/lib/aspera/persistency_action_once.rb +13 -14
  79. data/lib/aspera/persistency_folder.rb +5 -4
  80. data/lib/aspera/preview/file_types.rb +56 -268
  81. data/lib/aspera/preview/generator.rb +28 -39
  82. data/lib/aspera/preview/options.rb +2 -0
  83. data/lib/aspera/preview/terminal.rb +36 -16
  84. data/lib/aspera/preview/utils.rb +23 -29
  85. data/lib/aspera/proxy_auto_config.rb +6 -3
  86. data/lib/aspera/rest.rb +127 -80
  87. data/lib/aspera/rest_call_error.rb +1 -1
  88. data/lib/aspera/rest_error_analyzer.rb +16 -14
  89. data/lib/aspera/rest_errors_aspera.rb +39 -34
  90. data/lib/aspera/secret_hider.rb +18 -17
  91. data/lib/aspera/ssh.rb +10 -5
  92. data/lib/aspera/temp_file_manager.rb +11 -4
  93. data/lib/aspera/web_auth.rb +10 -7
  94. data/lib/aspera/web_server_simple.rb +11 -5
  95. data.tar.gz.sig +0 -0
  96. metadata +108 -39
  97. metadata.gz.sig +0 -0
  98. data/lib/aspera/cli/listener/line_dump.rb +0 -19
  99. data/lib/aspera/cli/listener/logger.rb +0 -22
  100. data/lib/aspera/cli/listener/progress.rb +0 -50
  101. data/lib/aspera/cli/listener/progress_multi.rb +0 -84
  102. data/lib/aspera/cli/plugins/sync.rb +0 -44
  103. data/lib/aspera/fasp/listener.rb +0 -13
  104. data/lib/aspera/sync.rb +0 -213
data/CONTRIBUTING.md CHANGED
@@ -1,5 +1,5 @@
1
1
  # Contributing
2
- <!-- cspell:words passin GEMVERS -->
2
+ <!-- cspell:words passin -->
3
3
  ## Reporting Issues and Vulnerabilities
4
4
 
5
5
  You can report issues at <https://github.com/IBM/aspera-cli/issues>
@@ -32,7 +32,7 @@ If you want to contribute, please:
32
32
  - Add tests for it. This is important so I don't break it in a future version unintentionally.
33
33
  - **Bonus Points** go out to anyone who also updates `CHANGELOG.md` :)
34
34
  - Send a pull request on GitHub.
35
- - run Rubocop to comply for coding standards
35
+ - run `rubocop` to comply for coding standards
36
36
 
37
37
  ## Architecture
38
38
 
@@ -42,32 +42,81 @@ Architecture:
42
42
 
43
43
  ![Architecture](docs/architecture.png)
44
44
 
45
- The entry point is lib/aspera/cli/main.rb
45
+ The entry point is: `lib/aspera/cli/main.rb`.
46
46
 
47
- Plugins are located in lib/aspera/cli/plugins
47
+ Plugins are located in: `lib/aspera/cli/plugins`.
48
48
 
49
- Transfer agents: lib/aspera/fasp
49
+ Transfer agents, in: `lib/aspera/fasp`.
50
+
51
+ ## Ruby version
52
+
53
+ Install ruby with [RVM](https://rvm.io).
54
+
55
+ Cleanup installed gems in `rvm` to start fresh:
56
+
57
+ ```bash
58
+ make cleanup_gems
59
+ ```
60
+
61
+ ## Tool chain
62
+
63
+ TODO: document installation of tool chain.
64
+
65
+ ### Documentation
66
+
67
+ Documentation is generated with `pandoc` and `LaTeX`.
68
+ IBM font `Plex` is used, see [IBM Plex](https://www.ibm.com/plex/).
50
69
 
51
70
  ## Running Tests
52
71
 
53
- First, a testing environment must be created:
72
+ First, a testing configuration file must be created, from project top folder:
54
73
 
55
74
  ```bash
56
- mkdir local
57
- cp docs/test_env.conf local/.
75
+ mkdir ~/some_secure_folder
76
+ cp docs/test_env.conf ~/some_secure_folder/.
58
77
  ```
59
78
 
60
- Fill `local/test_env.conf` with system URLs and credentials for tests.
79
+ Fill `~/some_secure_folder/test_env.conf` with system URLs and credentials for tests.
61
80
 
62
- This project uses a Makefile for tests:
81
+ Then tell where this file is located:
63
82
 
64
83
  ```bash
65
- cd tests
66
- make
84
+ export ASPERA_CLI_TEST_CONF_FILE=~/some_secure_folder/test_env.conf
85
+ ```
86
+
87
+ This project uses a `Makefile` for tests:
88
+
89
+ ```bash
90
+ make test
67
91
  ```
68
92
 
69
93
  When new commands are added to the CLI, new tests shall be added to the test suite.
70
94
 
95
+ ### Special tests
96
+
97
+ Some gems are optional: `rmagick` and `grpc`, as they require compilation of native code which may cause problems.
98
+ By default, tests that use those gems are skipped.
99
+ To run them: `make optional`.
100
+ Those tests also require the optional gems to be installed: `make install_optional_gems`.
101
+
102
+ Some other tests require interactive input. To run them: `make interactive`
103
+
104
+ To run every test: `make full`
105
+
106
+ ### Pre-release tests
107
+
108
+ For preparation of a release, do the following:
109
+
110
+ 1. Select a ruby version to test with: `rvm use 3.2.2`
111
+ 2. Remove all gems: `make clean_gems`
112
+ 3. Starts tests: `make test`
113
+ 4. Install optional gems: `make install_optional_gems`
114
+ 5. Test remaining cases: `cd tests` and then `make optional` and `make interactive`
115
+
116
+ This ensures that optional gems do not prevent `ascli` to run.
117
+
118
+ To test additional Ruby version, repeat the procedure with other Ruby versions.
119
+
71
120
  ## Coverage
72
121
 
73
122
  A coverage report can be generated in folder `coverage` using gem `SimpleCov`.
@@ -102,7 +151,7 @@ Then procedure is as follows:
102
151
  gem cert --build maintainer@example.com
103
152
  ```
104
153
 
105
- > **Note:** the email must match the field `spec.email` in `aspera-cli.gemspec`
154
+ > **Note:** The email must match the field `spec.email` in `aspera-cli.gemspec`
106
155
 
107
156
  This creates two files in folder `/path/to/vault` (e.g. $HOME/.ssh):
108
157
 
@@ -111,7 +160,7 @@ Then procedure is as follows:
111
160
 
112
161
  > **Note:** Alternatively, use an existing key or generate one, and then `make new-cert`
113
162
 
114
- - The maintainer build the signed gem.
163
+ - The maintainer builds the signed gem.
115
164
 
116
165
  The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
117
166
 
@@ -187,26 +236,28 @@ A template is used as it allows some level of customization to tell where to tak
187
236
  Then, to push to the image registry (both tags: version and `latest`):
188
237
 
189
238
  ```bash
190
- make dpush
239
+ make docker_push
191
240
  ```
192
241
 
193
242
  ### Specific version image build
194
243
 
195
- It is possible to build a specific version by setting envvar `GEMVERS` and `make` with option `-e`:
244
+ To build a specific version: set envvar `GEM_VERSION` and `make` with option `-e`:
196
245
 
197
246
  ```bash
198
- GEMVERS=4.11.0 make -e docker
199
- GEMVERS=4.11.0 make -e dpush
247
+ GEM_VERSION=4.11.0 make -e docker
248
+ GEM_VERSION=4.11.0 make -e docker_push
200
249
  ```
201
250
 
251
+ > **Note:** This does not use the gem locally, only the docker file is used locally, the gem is installed from rubygems.org.
252
+
202
253
  ### Development version image build
203
254
 
204
255
  To build/push a beta/development container:
205
- it does not create the `latest` tag, it uses the gem file generated locally.
256
+ it does not create the `latest` tag, it uses the gem file generated locally with a special version number.
206
257
 
207
258
  ```bash
208
- make dockerbeta
209
- make dpushversion
259
+ make docker_beta
260
+ make docker_push_beta
210
261
  ```
211
262
 
212
263
  ## Single executable build
@@ -219,7 +270,7 @@ make single
219
270
 
220
271
  This is based on tool `rubyc` (gem `ruby-packer`): either place the executable in folder `tmp`, or the architecture-specific version will be automatically downloaded.
221
272
 
222
- > **Note:** Perl Findbin is needed, install on Linux with:
273
+ > **Note:** Perl `Findbin` is needed, install on Linux with:
223
274
 
224
275
  ```bash
225
276
  sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
@@ -231,4 +282,4 @@ sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
231
282
  - <https://github.com/rest-client/rest-client>
232
283
  - <https://github.com/oauth-xx/oauth2>
233
284
  - use gem Thor <http://whatisthor.com/> (or other standard Ruby CLI manager)
234
- - Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (rubyc)
285
+ - Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (`rubyc`)