aspera-cli 4.15.0 → 4.17.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (108) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/BUGS.md +29 -3
  4. data/CHANGELOG.md +375 -280
  5. data/CONTRIBUTING.md +71 -18
  6. data/README.md +1978 -1656
  7. data/bin/ascli +13 -31
  8. data/bin/asession +32 -22
  9. data/examples/dascli +2 -2
  10. data/lib/aspera/agent/alpha.rb +117 -0
  11. data/lib/aspera/agent/base.rb +61 -0
  12. data/lib/aspera/{fasp/agent_connect.rb → agent/connect.rb} +13 -11
  13. data/lib/aspera/{fasp/agent_direct.rb → agent/direct.rb} +116 -116
  14. data/lib/aspera/{fasp/agent_httpgw.rb → agent/httpgw.rb} +21 -19
  15. data/lib/aspera/{fasp/agent_node.rb → agent/node.rb} +21 -33
  16. data/lib/aspera/agent/trsdk.rb +188 -0
  17. data/lib/aspera/api/aoc.rb +586 -0
  18. data/lib/aspera/api/ats.rb +46 -0
  19. data/lib/aspera/api/cos_node.rb +95 -0
  20. data/lib/aspera/api/node.rb +344 -0
  21. data/lib/aspera/ascmd.rb +47 -14
  22. data/lib/aspera/{fasp → ascp}/installation.rb +54 -15
  23. data/lib/aspera/{fasp → ascp}/management.rb +14 -14
  24. data/lib/aspera/{fasp → ascp}/products.rb +1 -1
  25. data/lib/aspera/assert.rb +45 -0
  26. data/lib/aspera/cli/basic_auth_plugin.rb +11 -10
  27. data/lib/aspera/cli/extended_value.rb +5 -5
  28. data/lib/aspera/cli/formatter.rb +27 -14
  29. data/lib/aspera/cli/hints.rb +7 -6
  30. data/lib/aspera/cli/main.rb +49 -29
  31. data/lib/aspera/cli/manager.rb +46 -36
  32. data/lib/aspera/cli/plugin.rb +34 -20
  33. data/lib/aspera/cli/plugin_factory.rb +61 -0
  34. data/lib/aspera/cli/plugins/alee.rb +7 -7
  35. data/lib/aspera/cli/plugins/aoc.rb +168 -132
  36. data/lib/aspera/cli/plugins/ats.rb +33 -33
  37. data/lib/aspera/cli/plugins/bss.rb +3 -4
  38. data/lib/aspera/cli/plugins/config.rb +250 -272
  39. data/lib/aspera/cli/plugins/console.rb +8 -6
  40. data/lib/aspera/cli/plugins/cos.rb +20 -19
  41. data/lib/aspera/cli/plugins/faspex.rb +71 -60
  42. data/lib/aspera/cli/plugins/faspex5.rb +212 -133
  43. data/lib/aspera/cli/plugins/node.rb +83 -75
  44. data/lib/aspera/cli/plugins/orchestrator.rb +36 -44
  45. data/lib/aspera/cli/plugins/preview.rb +33 -31
  46. data/lib/aspera/cli/plugins/server.rb +33 -32
  47. data/lib/aspera/cli/plugins/shares.rb +39 -33
  48. data/lib/aspera/cli/sync_actions.rb +9 -9
  49. data/lib/aspera/cli/transfer_agent.rb +45 -25
  50. data/lib/aspera/cli/transfer_progress.rb +2 -3
  51. data/lib/aspera/cli/version.rb +1 -1
  52. data/lib/aspera/colors.rb +5 -0
  53. data/lib/aspera/command_line_builder.rb +16 -14
  54. data/lib/aspera/coverage.rb +21 -0
  55. data/lib/aspera/data_repository.rb +33 -2
  56. data/lib/aspera/environment.rb +5 -4
  57. data/lib/aspera/faspex_gw.rb +13 -11
  58. data/lib/aspera/faspex_postproc.rb +6 -5
  59. data/lib/aspera/id_generator.rb +4 -2
  60. data/lib/aspera/json_rpc.rb +10 -8
  61. data/lib/aspera/keychain/encrypted_hash.rb +46 -11
  62. data/lib/aspera/keychain/macos_security.rb +29 -22
  63. data/lib/aspera/log.rb +5 -4
  64. data/lib/aspera/nagios.rb +7 -2
  65. data/lib/aspera/node_simulator.rb +213 -0
  66. data/lib/aspera/oauth/base.rb +143 -0
  67. data/lib/aspera/oauth/factory.rb +124 -0
  68. data/lib/aspera/oauth/generic.rb +34 -0
  69. data/lib/aspera/oauth/jwt.rb +51 -0
  70. data/lib/aspera/oauth/url_json.rb +31 -0
  71. data/lib/aspera/oauth/web.rb +50 -0
  72. data/lib/aspera/oauth.rb +5 -328
  73. data/lib/aspera/open_application.rb +7 -7
  74. data/lib/aspera/persistency_action_once.rb +13 -14
  75. data/lib/aspera/persistency_folder.rb +3 -2
  76. data/lib/aspera/preview/file_types.rb +53 -267
  77. data/lib/aspera/preview/generator.rb +7 -5
  78. data/lib/aspera/preview/terminal.rb +17 -7
  79. data/lib/aspera/preview/utils.rb +8 -7
  80. data/lib/aspera/proxy_auto_config.rb +6 -3
  81. data/lib/aspera/rest.rb +187 -140
  82. data/lib/aspera/rest_error_analyzer.rb +1 -0
  83. data/lib/aspera/rest_errors_aspera.rb +5 -3
  84. data/lib/aspera/resumer.rb +77 -0
  85. data/lib/aspera/secret_hider.rb +5 -2
  86. data/lib/aspera/ssh.rb +15 -8
  87. data/lib/aspera/temp_file_manager.rb +1 -1
  88. data/lib/aspera/{fasp → transfer}/error.rb +3 -3
  89. data/lib/aspera/{fasp → transfer}/error_info.rb +1 -1
  90. data/lib/aspera/{fasp → transfer}/faux_file.rb +1 -1
  91. data/lib/aspera/{fasp → transfer}/parameters.rb +95 -120
  92. data/lib/aspera/{fasp/transfer_spec.rb → transfer/spec.rb} +23 -19
  93. data/lib/aspera/{fasp/parameters.yaml → transfer/spec.yaml} +4 -99
  94. data/lib/aspera/transfer/sync.rb +273 -0
  95. data/lib/aspera/{fasp → transfer}/uri.rb +10 -9
  96. data/lib/aspera/web_server_simple.rb +12 -3
  97. data.tar.gz.sig +0 -0
  98. metadata +92 -68
  99. metadata.gz.sig +0 -0
  100. data/lib/aspera/aoc.rb +0 -606
  101. data/lib/aspera/ats_api.rb +0 -47
  102. data/lib/aspera/cos_node.rb +0 -93
  103. data/lib/aspera/fasp/agent_aspera.rb +0 -126
  104. data/lib/aspera/fasp/agent_base.rb +0 -48
  105. data/lib/aspera/fasp/agent_trsdk.rb +0 -146
  106. data/lib/aspera/fasp/resume_policy.rb +0 -77
  107. data/lib/aspera/node.rb +0 -338
  108. data/lib/aspera/sync.rb +0 -219
data/CONTRIBUTING.md CHANGED
@@ -42,32 +42,81 @@ Architecture:
42
42
 
43
43
  ![Architecture](docs/architecture.png)
44
44
 
45
- The entry point is lib/aspera/cli/main.rb
45
+ The entry point is: `lib/aspera/cli/main.rb`.
46
46
 
47
- Plugins are located in lib/aspera/cli/plugins
47
+ Plugins are located in: `lib/aspera/cli/plugins`.
48
48
 
49
- Transfer agents: lib/aspera/fasp
49
+ Transfer agents, in: `lib/aspera/fasp`.
50
+
51
+ ## Ruby version
52
+
53
+ Install ruby with [RVM](https://rvm.io).
54
+
55
+ Cleanup installed gems in `rvm` to start fresh:
56
+
57
+ ```bash
58
+ make cleanup_gems
59
+ ```
60
+
61
+ ## Tool chain
62
+
63
+ TODO: document installation of tool chain.
64
+
65
+ ### Documentation
66
+
67
+ Documentation is generated with `pandoc` and `LaTeX`.
68
+ IBM font `Plex` is used, see [IBM Plex](https://www.ibm.com/plex/).
50
69
 
51
70
  ## Running Tests
52
71
 
53
- First, a testing environment must be created:
72
+ First, a testing configuration file must be created, from project top folder:
54
73
 
55
74
  ```bash
56
- mkdir local
57
- cp docs/test_env.conf local/.
75
+ mkdir ~/some_secure_folder
76
+ cp docs/test_env.conf ~/some_secure_folder/.
58
77
  ```
59
78
 
60
- Fill `local/test_env.conf` with system URLs and credentials for tests.
79
+ Fill `~/some_secure_folder/test_env.conf` with system URLs and credentials for tests.
61
80
 
62
- This project uses a Makefile for tests:
81
+ Then tell where this file is located:
63
82
 
64
83
  ```bash
65
- cd tests
66
- make
84
+ export ASPERA_CLI_TEST_CONF_FILE=~/some_secure_folder/test_env.conf
85
+ ```
86
+
87
+ This project uses a `Makefile` for tests:
88
+
89
+ ```bash
90
+ make test
67
91
  ```
68
92
 
69
93
  When new commands are added to the CLI, new tests shall be added to the test suite.
70
94
 
95
+ ### Special tests
96
+
97
+ Some gems are optional: `rmagick` and `grpc`, as they require compilation of native code which may cause problems.
98
+ By default, tests that use those gems are skipped.
99
+ To run them: `make optional`.
100
+ Those tests also require the optional gems to be installed: `make install_optional_gems`.
101
+
102
+ Some other tests require interactive input. To run them: `make interactive`
103
+
104
+ To run every test: `make full`
105
+
106
+ ### Pre-release tests
107
+
108
+ For preparation of a release, do the following:
109
+
110
+ 1. Select a ruby version to test with: `rvm use 3.2.2`
111
+ 2. Remove all gems: `make clean_gems`
112
+ 3. Starts tests: `make test`
113
+ 4. Install optional gems: `make install_optional_gems`
114
+ 5. Test remaining cases: `cd tests` and then `make optional` and `make interactive`
115
+
116
+ This ensures that optional gems do not prevent `ascli` to run.
117
+
118
+ To test additional Ruby version, repeat the procedure with other Ruby versions.
119
+
71
120
  ## Coverage
72
121
 
73
122
  A coverage report can be generated in folder `coverage` using gem `SimpleCov`.
@@ -102,7 +151,7 @@ Then procedure is as follows:
102
151
  gem cert --build maintainer@example.com
103
152
  ```
104
153
 
105
- > **Note:** the email must match the field `spec.email` in `aspera-cli.gemspec`
154
+ > **Note:** The email must match the field `spec.email` in `aspera-cli.gemspec`
106
155
 
107
156
  This creates two files in folder `/path/to/vault` (e.g. $HOME/.ssh):
108
157
 
@@ -111,7 +160,7 @@ Then procedure is as follows:
111
160
 
112
161
  > **Note:** Alternatively, use an existing key or generate one, and then `make new-cert`
113
162
 
114
- - The maintainer build the signed gem.
163
+ - The maintainer builds the signed gem.
115
164
 
116
165
  The gem is signed with the public certificate found in `certs` and the private key (kept secret by maintainer).
117
166
 
@@ -187,26 +236,30 @@ A template is used as it allows some level of customization to tell where to tak
187
236
  Then, to push to the image registry (both tags: version and `latest`):
188
237
 
189
238
  ```bash
190
- make dpush
239
+ make docker_push
191
240
  ```
192
241
 
193
242
  ### Specific version image build
194
243
 
195
- It is possible to build a specific version by setting envvar `GEM_VERSION` and `make` with option `-e`:
244
+ To build a specific version: set envvar `GEM_VERSION` and `make` with option `-e`:
196
245
 
197
246
  ```bash
198
247
  GEM_VERSION=4.11.0 make -e docker
199
- GEM_VERSION=4.11.0 make -e dpush
248
+ GEM_VERSION=4.11.0 make -e docker_push
200
249
  ```
201
250
 
251
+ > **Note:** This does not use the locally generated gem file.
252
+ Only the local docker file is used.
253
+ The gem is installed from rubygems.org. This also sets the `latest` tag.
254
+
202
255
  ### Development version image build
203
256
 
204
257
  To build/push a beta/development container:
205
- it does not create the `latest` tag, it uses the gem file generated locally.
258
+ it does not create the `latest` tag, it uses the gem file generated locally with a special version number.
206
259
 
207
260
  ```bash
208
- make dockerbeta
209
- make dpushversion
261
+ make docker_beta
262
+ make docker_push_beta
210
263
  ```
211
264
 
212
265
  ## Single executable build