aspera-cli 4.14.0 → 4.16.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (104) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/BUGS.md +29 -3
  4. data/CHANGELOG.md +300 -185
  5. data/CONTRIBUTING.md +74 -23
  6. data/README.md +2346 -1619
  7. data/bin/ascli +16 -25
  8. data/bin/asession +15 -15
  9. data/examples/dascli +2 -2
  10. data/examples/proxy.pac +1 -1
  11. data/lib/aspera/aoc.rb +216 -150
  12. data/lib/aspera/ascmd.rb +25 -18
  13. data/lib/aspera/assert.rb +45 -0
  14. data/lib/aspera/cli/basic_auth_plugin.rb +9 -6
  15. data/lib/aspera/cli/error.rb +17 -0
  16. data/lib/aspera/cli/extended_value.rb +51 -16
  17. data/lib/aspera/cli/formatter.rb +276 -174
  18. data/lib/aspera/cli/hints.rb +81 -0
  19. data/lib/aspera/cli/main.rb +114 -147
  20. data/lib/aspera/cli/manager.rb +181 -136
  21. data/lib/aspera/cli/plugin.rb +82 -64
  22. data/lib/aspera/cli/plugins/alee.rb +0 -1
  23. data/lib/aspera/cli/plugins/aoc.rb +327 -331
  24. data/lib/aspera/cli/plugins/ats.rb +12 -8
  25. data/lib/aspera/cli/plugins/bss.rb +2 -2
  26. data/lib/aspera/cli/plugins/config.rb +575 -439
  27. data/lib/aspera/cli/plugins/console.rb +40 -0
  28. data/lib/aspera/cli/plugins/cos.rb +4 -5
  29. data/lib/aspera/cli/plugins/faspex.rb +111 -92
  30. data/lib/aspera/cli/plugins/faspex5.rb +245 -182
  31. data/lib/aspera/cli/plugins/node.rb +239 -160
  32. data/lib/aspera/cli/plugins/orchestrator.rb +56 -19
  33. data/lib/aspera/cli/plugins/preview.rb +54 -38
  34. data/lib/aspera/cli/plugins/server.rb +63 -20
  35. data/lib/aspera/cli/plugins/shares.rb +64 -38
  36. data/lib/aspera/cli/sync_actions.rb +68 -0
  37. data/lib/aspera/cli/transfer_agent.rb +64 -67
  38. data/lib/aspera/cli/transfer_progress.rb +73 -0
  39. data/lib/aspera/cli/version.rb +1 -1
  40. data/lib/aspera/colors.rb +3 -1
  41. data/lib/aspera/command_line_builder.rb +27 -22
  42. data/lib/aspera/cos_node.rb +6 -4
  43. data/lib/aspera/coverage.rb +22 -0
  44. data/lib/aspera/data_repository.rb +33 -2
  45. data/lib/aspera/environment.rb +21 -8
  46. data/lib/aspera/fasp/agent_alpha.rb +116 -0
  47. data/lib/aspera/fasp/agent_base.rb +40 -76
  48. data/lib/aspera/fasp/agent_connect.rb +21 -22
  49. data/lib/aspera/fasp/agent_direct.rb +169 -179
  50. data/lib/aspera/fasp/agent_httpgw.rb +200 -195
  51. data/lib/aspera/fasp/agent_node.rb +43 -35
  52. data/lib/aspera/fasp/agent_trsdk.rb +124 -41
  53. data/lib/aspera/fasp/error_info.rb +2 -2
  54. data/lib/aspera/fasp/faux_file.rb +52 -0
  55. data/lib/aspera/fasp/installation.rb +89 -191
  56. data/lib/aspera/fasp/management.rb +249 -0
  57. data/lib/aspera/fasp/parameters.rb +86 -47
  58. data/lib/aspera/fasp/parameters.yaml +75 -8
  59. data/lib/aspera/fasp/products.rb +162 -0
  60. data/lib/aspera/fasp/resume_policy.rb +7 -5
  61. data/lib/aspera/fasp/sync.rb +273 -0
  62. data/lib/aspera/fasp/transfer_spec.rb +10 -8
  63. data/lib/aspera/fasp/uri.rb +6 -6
  64. data/lib/aspera/faspex_gw.rb +11 -8
  65. data/lib/aspera/faspex_postproc.rb +8 -7
  66. data/lib/aspera/hash_ext.rb +2 -2
  67. data/lib/aspera/id_generator.rb +3 -1
  68. data/lib/aspera/json_rpc.rb +51 -0
  69. data/lib/aspera/keychain/encrypted_hash.rb +46 -11
  70. data/lib/aspera/keychain/macos_security.rb +15 -13
  71. data/lib/aspera/line_logger.rb +23 -0
  72. data/lib/aspera/log.rb +61 -19
  73. data/lib/aspera/nagios.rb +7 -2
  74. data/lib/aspera/node.rb +105 -21
  75. data/lib/aspera/node_simulator.rb +214 -0
  76. data/lib/aspera/oauth.rb +57 -36
  77. data/lib/aspera/open_application.rb +4 -4
  78. data/lib/aspera/persistency_action_once.rb +13 -14
  79. data/lib/aspera/persistency_folder.rb +5 -4
  80. data/lib/aspera/preview/file_types.rb +56 -268
  81. data/lib/aspera/preview/generator.rb +28 -39
  82. data/lib/aspera/preview/options.rb +2 -0
  83. data/lib/aspera/preview/terminal.rb +36 -16
  84. data/lib/aspera/preview/utils.rb +23 -29
  85. data/lib/aspera/proxy_auto_config.rb +6 -3
  86. data/lib/aspera/rest.rb +127 -80
  87. data/lib/aspera/rest_call_error.rb +1 -1
  88. data/lib/aspera/rest_error_analyzer.rb +16 -14
  89. data/lib/aspera/rest_errors_aspera.rb +39 -34
  90. data/lib/aspera/secret_hider.rb +18 -17
  91. data/lib/aspera/ssh.rb +10 -5
  92. data/lib/aspera/temp_file_manager.rb +11 -4
  93. data/lib/aspera/web_auth.rb +10 -7
  94. data/lib/aspera/web_server_simple.rb +11 -5
  95. data.tar.gz.sig +0 -0
  96. metadata +108 -39
  97. metadata.gz.sig +0 -0
  98. data/lib/aspera/cli/listener/line_dump.rb +0 -19
  99. data/lib/aspera/cli/listener/logger.rb +0 -22
  100. data/lib/aspera/cli/listener/progress.rb +0 -50
  101. data/lib/aspera/cli/listener/progress_multi.rb +0 -84
  102. data/lib/aspera/cli/plugins/sync.rb +0 -44
  103. data/lib/aspera/fasp/listener.rb +0 -13
  104. data/lib/aspera/sync.rb +0 -213
data/CONTRIBUTING.md CHANGED
@@ -1,5 +1,5 @@
1
1
  # Contributing
2
- <!-- cspell:words passin GEMVERS -->
2
+ <!-- cspell:words passin -->
3
3
  ## Reporting Issues and Vulnerabilities
4
4
 
5
5
  You can report issues at <https://github.com/IBM/aspera-cli/issues>
@@ -32,7 +32,7 @@ If you want to contribute, please:
32
32
  - Add tests for it. This is important so I don't break it in a future version unintentionally.
33
33
  - **Bonus Points** go out to anyone who also updates `CHANGELOG.md` :)
34
34
  - Send a pull request on GitHub.
35
- - run Rubocop to comply for coding standards
35
+ - run `rubocop` to comply for coding standards
36
36
 
37
37
  ## Architecture
38
38
 
@@ -42,32 +42,81 @@ Architecture:
42
42
 
43
43
  ![Architecture](docs/architecture.png)
44
44
 
45
- The entry point is lib/aspera/cli/main.rb
45
+ The entry point is: `lib/aspera/cli/main.rb`.
46
46
 
47
- Plugins are located in lib/aspera/cli/plugins
47
+ Plugins are located in: `lib/aspera/cli/plugins`.
48
48
 
49
- Transfer agents: lib/aspera/fasp
49
+ Transfer agents, in: `lib/aspera/fasp`.
50
+
51
+ ## Ruby version
52
+
53
+ Install ruby with [RVM](https://rvm.io).
54
+
55
+ Cleanup installed gems in `rvm` to start fresh:
56
+
57
+ ```bash
58
+ make cleanup_gems
59
+ ```
60
+
61
+ ## Tool chain
62
+
63
+ TODO: document installation of tool chain.
64
+
65
+ ### Documentation
66
+
67
+ Documentation is generated with `pandoc` and `LaTeX`.
68
+ IBM font `Plex` is used, see [IBM Plex](https://www.ibm.com/plex/).
50
69
 
51
70
  ## Running Tests
52
71
 
53
- First, a testing environment must be created:
72
+ First, a testing configuration file must be created, from project top folder:
54
73
 
55
74
  ```bash
56
- mkdir local
57
- cp docs/test_env.conf local/.
75
+ mkdir ~/some_secure_folder
76
+ cp docs/test_env.conf ~/some_secure_folder/.
58
77
  ```
59
78
 
60
- Fill `local/test_env.conf` with system URLs and credentials for tests.
79
+ Fill `~/some_secure_folder/test_env.conf` with system URLs and credentials for tests.
61
80
 
62
- This project uses a Makefile for tests:
81
+ Then tell where this file is located:
63
82
 
64
83
  ```bash
65
- cd tests
66
- make
84
+ export ASPERA_CLI_TEST_CONF_FILE=~/some_secure_folder/test_env.conf
85
+ ```
86
+
87
+ This project uses a `Makefile` for tests:
88
+
89
+ ```bash
90
+ make test
67
91
  ```
68
92
 
69
93
  When new commands are added to the CLI, new tests shall be added to the test suite.
70
94
 
95
+ ### Special tests
96
+
97
+ Some gems are optional: `rmagick` and `grpc`, as they require compilation of native code which may cause problems.
98
+ By default, tests that use those gems are skipped.
99
+ To run them: `make optional`.
100
+ Those tests also require the optional gems to be installed: `make install_optional_gems`.
101
+
102
+ Some other tests require interactive input. To run them: `make interactive`
103
+
104
+ To run every test: `make full`
105
+
106
+ ### Pre-release tests
107
+
108
+ For preparation of a release, do the following:
109
+
110
+ 1. Select a ruby version to test with: `rvm use 3.2.2`
111
+ 2. Remove all gems: `make clean_gems`
112
+ 3. Starts tests: `make test`
113
+ 4. Install optional gems: `make install_optional_gems`
114
+ 5. Test remaining cases: `cd tests` and then `make optional` and `make interactive`
115
+
116
+ This ensures that optional gems do not prevent `ascli` to run.
117
+
118
+ To test additional Ruby version, repeat the procedure with other Ruby versions.
119
+
71
120
  ## Coverage
72
121
 
73
122
  A coverage report can be generated in folder `coverage` using gem `SimpleCov`.
@@ -102,7 +151,7 @@ Then procedure is as follows:
102
151
  gem cert --build maintainer@example.com
103
152
  ```
104
153
 
105
- > **Note:** the email must match the field `spec.email` in `aspera-cli.gemspec`
154
+ > **Note:** The email must match the field `spec.email` in `aspera-cli.gemspec`
106
155
 
107
156
  This creates two files in folder `/path/to/vault` (e.g. $HOME/.ssh):
108
157
 
@@ -111,7 +160,7 @@ Then procedure is as follows:
111
160
 
112
161
  > **Note:** Alternatively, use an existing key or generate one, and then `make new-cert`
113
162
 
114
- - The maintainer build the signed gem.
163
+ - The maintainer builds the signed gem.
115
164
 
116
165
  The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
117
166
 
@@ -187,26 +236,28 @@ A template is used as it allows some level of customization to tell where to tak
187
236
  Then, to push to the image registry (both tags: version and `latest`):
188
237
 
189
238
  ```bash
190
- make dpush
239
+ make docker_push
191
240
  ```
192
241
 
193
242
  ### Specific version image build
194
243
 
195
- It is possible to build a specific version by setting envvar `GEMVERS` and `make` with option `-e`:
244
+ To build a specific version: set envvar `GEM_VERSION` and `make` with option `-e`:
196
245
 
197
246
  ```bash
198
- GEMVERS=4.11.0 make -e docker
199
- GEMVERS=4.11.0 make -e dpush
247
+ GEM_VERSION=4.11.0 make -e docker
248
+ GEM_VERSION=4.11.0 make -e docker_push
200
249
  ```
201
250
 
251
+ > **Note:** This does not use the gem locally, only the docker file is used locally, the gem is installed from rubygems.org.
252
+
202
253
  ### Development version image build
203
254
 
204
255
  To build/push a beta/development container:
205
- it does not create the `latest` tag, it uses the gem file generated locally.
256
+ it does not create the `latest` tag, it uses the gem file generated locally with a special version number.
206
257
 
207
258
  ```bash
208
- make dockerbeta
209
- make dpushversion
259
+ make docker_beta
260
+ make docker_push_beta
210
261
  ```
211
262
 
212
263
  ## Single executable build
@@ -219,7 +270,7 @@ make single
219
270
 
220
271
  This is based on tool `rubyc` (gem `ruby-packer`): either place the executable in folder `tmp`, or the architecture-specific version will be automatically downloaded.
221
272
 
222
- > **Note:** Perl Findbin is needed, install on Linux with:
273
+ > **Note:** Perl `Findbin` is needed, install on Linux with:
223
274
 
224
275
  ```bash
225
276
  sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
@@ -231,4 +282,4 @@ sudo dnf install -y perl perl-FindBin perl-IPC-Cmd
231
282
  - <https://github.com/rest-client/rest-client>
232
283
  - <https://github.com/oauth-xx/oauth2>
233
284
  - use gem Thor <http://whatisthor.com/> (or other standard Ruby CLI manager)
234
- - Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (rubyc)
285
+ - Package a single-file executable for various architectures with <https://github.com/pmq20/ruby-packer> (`rubyc`)