aspera-cli 4.23.0 → 4.24.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (109) hide show
  1. checksums.yaml +4 -4
  2. checksums.yaml.gz.sig +0 -0
  3. data/CHANGELOG.md +32 -1
  4. data/CONTRIBUTING.md +86 -29
  5. data/README.md +1651 -856
  6. data/bin/ascli +2 -1
  7. data/bin/asession +4 -4
  8. data/lib/aspera/agent/base.rb +4 -0
  9. data/lib/aspera/agent/connect.rb +20 -18
  10. data/lib/aspera/agent/desktop.rb +14 -11
  11. data/lib/aspera/agent/direct.rb +39 -31
  12. data/lib/aspera/agent/httpgw.rb +2 -2
  13. data/lib/aspera/agent/node.rb +9 -11
  14. data/lib/aspera/agent/transferd.rb +18 -11
  15. data/lib/aspera/api/aoc.rb +44 -31
  16. data/lib/aspera/api/cos_node.rb +7 -5
  17. data/lib/aspera/api/httpgw.rb +15 -18
  18. data/lib/aspera/api/node.rb +104 -22
  19. data/lib/aspera/ascmd.rb +22 -16
  20. data/lib/aspera/ascp/installation.rb +37 -40
  21. data/lib/aspera/ascp/management.rb +5 -4
  22. data/lib/aspera/assert.rb +54 -23
  23. data/lib/aspera/cli/basic_auth_plugin.rb +8 -7
  24. data/lib/aspera/cli/error.rb +1 -1
  25. data/lib/aspera/cli/extended_value.rb +28 -29
  26. data/lib/aspera/cli/formatter.rb +191 -168
  27. data/lib/aspera/cli/hints.rb +29 -3
  28. data/lib/aspera/cli/main.rb +138 -107
  29. data/lib/aspera/cli/manager.rb +50 -30
  30. data/lib/aspera/cli/plugin.rb +148 -77
  31. data/lib/aspera/cli/plugin_factory.rb +2 -2
  32. data/lib/aspera/cli/plugins/aoc.rb +189 -70
  33. data/lib/aspera/cli/plugins/ats.rb +15 -13
  34. data/lib/aspera/cli/plugins/config.rb +86 -213
  35. data/lib/aspera/cli/plugins/console.rb +49 -18
  36. data/lib/aspera/cli/plugins/cos.rb +4 -4
  37. data/lib/aspera/cli/plugins/faspex.rb +45 -51
  38. data/lib/aspera/cli/plugins/faspex5.rb +162 -163
  39. data/lib/aspera/cli/plugins/faspio.rb +6 -5
  40. data/lib/aspera/cli/plugins/httpgw.rb +2 -2
  41. data/lib/aspera/cli/plugins/node.rb +144 -162
  42. data/lib/aspera/cli/plugins/orchestrator.rb +10 -14
  43. data/lib/aspera/cli/plugins/preview.rb +26 -29
  44. data/lib/aspera/cli/plugins/server.rb +28 -28
  45. data/lib/aspera/cli/plugins/shares.rb +40 -28
  46. data/lib/aspera/cli/sync_actions.rb +101 -80
  47. data/lib/aspera/cli/transfer_agent.rb +51 -50
  48. data/lib/aspera/cli/transfer_progress.rb +29 -20
  49. data/lib/aspera/cli/version.rb +1 -1
  50. data/lib/aspera/cli/wizard.rb +160 -0
  51. data/lib/aspera/colors.rb +13 -8
  52. data/lib/aspera/command_line_builder.rb +28 -22
  53. data/lib/aspera/command_line_converter.rb +31 -0
  54. data/lib/aspera/environment.rb +144 -101
  55. data/lib/aspera/faspex_gw.rb +1 -1
  56. data/lib/aspera/faspex_postproc.rb +3 -2
  57. data/lib/aspera/hash_ext.rb +1 -1
  58. data/lib/aspera/id_generator.rb +10 -10
  59. data/lib/aspera/keychain/base.rb +18 -0
  60. data/lib/aspera/keychain/encrypted_hash.rb +6 -12
  61. data/lib/aspera/keychain/factory.rb +9 -3
  62. data/lib/aspera/keychain/hashicorp_vault.rb +9 -6
  63. data/lib/aspera/keychain/macos_security.rb +13 -13
  64. data/lib/aspera/log.rb +69 -20
  65. data/lib/aspera/nagios.rb +5 -6
  66. data/lib/aspera/node_simulator.rb +12 -7
  67. data/lib/aspera/oauth/base.rb +5 -3
  68. data/lib/aspera/oauth/factory.rb +24 -18
  69. data/lib/aspera/oauth/jwt.rb +13 -1
  70. data/lib/aspera/oauth/url_json.rb +3 -3
  71. data/lib/aspera/oauth/web.rb +5 -3
  72. data/lib/aspera/persistency_folder.rb +2 -2
  73. data/lib/aspera/preview/file_types.rb +4 -3
  74. data/lib/aspera/preview/generator.rb +25 -12
  75. data/lib/aspera/preview/terminal.rb +10 -7
  76. data/lib/aspera/preview/utils.rb +11 -9
  77. data/lib/aspera/products/connect.rb +1 -1
  78. data/lib/aspera/products/desktop.rb +1 -1
  79. data/lib/aspera/products/other.rb +2 -2
  80. data/lib/aspera/products/transferd.rb +8 -6
  81. data/lib/aspera/proxy_auto_config.rb +1 -1
  82. data/lib/aspera/rest.rb +29 -22
  83. data/lib/aspera/rest_call_error.rb +1 -1
  84. data/lib/aspera/resumer.rb +1 -1
  85. data/lib/aspera/secret_hider.rb +46 -40
  86. data/lib/aspera/ssh.rb +13 -3
  87. data/lib/aspera/sync/args.schema.yaml +102 -0
  88. data/lib/aspera/sync/conf.schema.yaml +701 -0
  89. data/lib/aspera/sync/database.rb +83 -0
  90. data/lib/aspera/{transfer/sync.rb → sync/operations.rb} +132 -65
  91. data/lib/aspera/temp_file_manager.rb +3 -2
  92. data/lib/aspera/transfer/error.rb +1 -1
  93. data/lib/aspera/transfer/error_info.rb +1 -2
  94. data/lib/aspera/transfer/faux_file.rb +11 -10
  95. data/lib/aspera/transfer/parameters.rb +6 -5
  96. data/lib/aspera/transfer/spec.rb +15 -1
  97. data/lib/aspera/transfer/spec.schema.yaml +316 -293
  98. data/lib/aspera/transfer/spec_doc.rb +34 -16
  99. data/lib/aspera/transfer/uri.rb +5 -5
  100. data/lib/aspera/uri_reader.rb +14 -10
  101. data/lib/aspera/web_auth.rb +2 -2
  102. data/lib/aspera/web_server_simple.rb +2 -2
  103. data.tar.gz.sig +0 -0
  104. metadata +15 -13
  105. metadata.gz.sig +2 -2
  106. data/lib/aspera/transfer/async_conf.schema.yaml +0 -716
  107. data/lib/aspera/transfer/convert.rb +0 -29
  108. data/lib/aspera/transfer/sync_instance.schema.yaml +0 -20
  109. data/lib/aspera/transfer/sync_session.schema.yaml +0 -86
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 7e7c12c376e7d332f07acaf7632a095c31610170103b27f96eea43667f4fa3b3
4
- data.tar.gz: 84f9f1b90a08665b94e0d23016d74b7b54fc1b53855dba544237e908cc3fc49e
3
+ metadata.gz: f82c20bd2d1eb71e5942cbd45eed6e983770ee8867bd0e403325a1f762e0b667
4
+ data.tar.gz: d77e927f20370db6dfad42683b74e5fc9e24dea70ed59da26704f911aa5d3361
5
5
  SHA512:
6
- metadata.gz: 9051ce885074c8df74dc4a8a19fc54463c6f9d1e954bef7f4634a198a443bdd615edd00c018e53d5a4066715b455fdaa0e5de6812b09dded3d1ec5b85a6f455f
7
- data.tar.gz: 66fbf7d28bde0998170b9288d6cca9f201a766da99d764c2a72b5ceb64f2d47b76eb48954392f21d10713dbee591e2622c167abe943e32a9529e4c971f8d4be8
6
+ metadata.gz: f756c074e054ca7ddecdc7fd9bf89e577e42f30480a7868bfda00a259c08d9362f1fc77abab5779cfd62ccdcd1bef067c74152d0f4435f92f0f166d031182f07
7
+ data.tar.gz: 92cdb44671b44ba5eff6312bfb2212cc6f15bc0d73d0cc9608c135e41ec4ee8ec68ccd486684b075fe03767918da2135e190bc9b018ef36ed93e9416713e2303
checksums.yaml.gz.sig CHANGED
Binary file
data/CHANGELOG.md CHANGED
@@ -1,6 +1,37 @@
1
1
  # Changes (Release notes)
2
2
 
3
+ * 4.24.0
4
+
5
+ * New Features:
6
+ * `aoc`: Option `package_folder` allows specification of secondary field.
7
+ * `config`: New option `invalid_characters` ensures generated file names are valid.
8
+ * `config`: Added support for **dot-separated** option names, allowing nested hash structures to be specified directly on the command line.
9
+ * `console`: Added support for extended filters in transfer queries.
10
+ * `http_options`: New field `ssl_options` allows setting SSL Context options.
11
+ * `format`: `csv` format now supports option `table_style` for customizable output.
12
+ * `logger`: New option: `log_format` to control log formatting.
13
+ * `sync`: New command: `db` for operations on sync database.
14
+ * `sync`: Sync operations now use options `ts` and `to_folder`.
15
+ * Issues Fixed:
16
+ * JRuby: Modified tests and documentation for special SSH options.
17
+ * `transferd`: Fixed discrepancies in transfer spec resume policies.
18
+ * `desktop`: Fixed discrepancies in transfer spec resume policies.
19
+ * `format`: Value of type `list` now properly display column headers.
20
+ * `select`: Filter is now done on values before enhanced display in table mode.
21
+ * `aoc`: #221 Fixed package encryption at rest (CSEAR) status.
22
+ * Breaking Changes:
23
+ * `config`: Option `silent_insecure` renamed `warn_insecure`. `yes` shows warning (default).
24
+ * `ts` : Default transfer spec includes `resume_policy=sparse_csum`.
25
+ * `ssh_options` : Now additive option, like `ts`.
26
+ * `vault`: When creating an entry, the `label` field is now part of the creation Hash.
27
+ * `console`: Removed options `filter_from` and `filter_to`. Use standard option `query` instead.
28
+ * `sync`: Removed option `sync_info`. Replaced with positional parameters. Streamlined command line interface. Applies to all plugins with `sync` command.
29
+ * `async`: Removed option `sync_name`. Replaced with percent selector `%name:`.
30
+ * `aoc`: `files download` using gen4 API do not require anymore to provide the containing folder in first position, and then only file names. Now, directly provide the path to all files.
31
+ * `logger`: Log is simplified, date is removed by default. Use `--log_format=standard` to revert to standard Ruby logger. See option `log_format` for details.
32
+
3
33
  * 4.23.0 2025-08-11
34
+
4
35
  * New Features:
5
36
  * `aoc`: #201: Added `package_folder` option to place each received package in its own subfolder named after a package attribute. Default is `@none:` which means no subfolder will be created.
6
37
  * `config`: Added `transferd` version 1.1.6.
@@ -182,7 +213,7 @@
182
213
  * `faspex5`: When receiving ALL packages, only get those with status `completed`.
183
214
  * `direct` agent: Better support for WSS
184
215
  * Breaking Changes:
185
- * `shares`: Option `type` for users and groups is replaced with mandatory positional argument with same value.
216
+ * `shares`: Option `type` for users and groups is replaced with mandatory positional argument with same value. E.g. `user list --type=local` becomes: `user local list`.
186
217
  * `aoc`, `faspex`: Package `recv` command changed to `receive`, for consistency with faspex5 (`recv` is now an alias command)
187
218
 
188
219
  * 4.15.0 2023-11-18
data/CONTRIBUTING.md CHANGED
@@ -2,62 +2,99 @@
2
2
  <!-- cspell:words passin -->
3
3
  ## Reporting Issues and Vulnerabilities
4
4
 
5
- You can report issues at <https://github.com/IBM/aspera-cli/issues>
5
+ If you encounter a problem or vulnerability, please report it at [GitHub Issues](https://github.com/IBM/aspera-cli/issues).
6
6
 
7
- Before you go ahead please search existing issues for your problem.
7
+ Before submitting a new issue:
8
8
 
9
- To make sure that we can help you quickly please include and check the following information:
9
+ - Search existing issues to see if your problem has already been reported or resolved.
10
10
 
11
- - Include the `ascli` version you are running in your report.
12
- - If you are not running the latest version (please check) then update.
13
- - Include your `ruby -e "puts RUBY_DESCRIPTION"`.
11
+ To help us assist you efficiently, include the following in your report:
14
12
 
15
- Thanks!
13
+ - The version of ascli you are using:
14
+
15
+ ```bash
16
+ ascli version
17
+ ```
18
+
19
+ - Confirmation that you're using the latest version (update if not).
20
+ - The output of your Ruby environment:
21
+
22
+ ```bash
23
+ ruby -e "puts RUBY_DESCRIPTION"
24
+ ```
16
25
 
17
26
  ## Making Contributions
18
27
 
19
- To fetch and test the gem for development, do:
28
+ We welcome contributions to improve the aspera-cli project!
29
+
30
+ ### Getting Started
31
+
32
+ Clone the repository and navigate to the project folder:
20
33
 
21
34
  ```bash
22
35
  git clone https://github.com/IBM/aspera-cli.git
23
36
  cd aspera-cli
24
37
  ```
25
38
 
26
- Then see [Running Tests](#running-tests).
39
+ For testing instructions, refer to [Running Tests](#running-tests).
27
40
 
28
- If you want to contribute, please:
41
+ ### How to Contribute
29
42
 
30
- - Fork the project.
31
- - Make your feature addition or bug fix.
32
- - Add tests for it. This is important, so it doesn't break in a future version unintentionally.
33
- - Run `rubocop` to comply for coding standards
34
- - Update `CHANGELOG.md`
35
- - Send a pull request on GitHub.
43
+ To submit a contribution:
36
44
 
37
- ## Architecture
45
+ 1. **Fork** the repository on GitHub.
38
46
 
39
- A list of classes are provided in <docs/uml.png>
47
+ 1. **Create a feature branch** for your changes.
48
+
49
+ 1. **Implement** your feature or bug fix.
50
+
51
+ 1. **Write tests** to ensure your changes are robust and won’t break future versions.
52
+
53
+ 1. Run `rubocop` to ensure your code follows the Ruby style guide.
54
+
55
+ 1. Update `CHANGELOG.md` with a summary of your changes.
56
+
57
+ 1. Submit a **pull request** with a clear description of your contribution.
58
+
59
+ > [!TIP]
60
+ > Make sure your pull request is focused and includes only relevant changes.
61
+
62
+ ## Architecture
40
63
 
41
- Architecture:
64
+ The overall architecture of aspera-cli is modular and extensible.
42
65
 
43
66
  ![Architecture](docs/architecture.png)
44
67
 
45
- The entry point is: `lib/aspera/cli/main.rb`.
68
+ Structure Highlights:
46
69
 
47
- Plugins are located in: `lib/aspera/cli/plugins`.
70
+ - Entry Point:
48
71
 
49
- Transfer agents, in: `lib/aspera/agent`.
72
+ `lib/aspera/cli/main.rb` - This is where the CLI execution begins.
73
+
74
+ - Plugins:
75
+
76
+ Located in `lib/aspera/cli/plugins`, plugins extend CLI functionality and encapsulate specific features.
77
+
78
+ - Transfer Agents:
79
+
80
+ Found in `lib/aspera/agent`, these handle data transfers operations.
81
+
82
+ A list of classes are provided in <docs/uml.png>
50
83
 
51
84
  ## Ruby version
52
85
 
53
- Install Ruby using your preferred method.
86
+ `aspera-cli` is built with Ruby.
87
+ You can install Ruby using any method you prefer (e.g., rbenv, rvm, system package manager).
54
88
 
55
- To clean up installed gems to start fresh:
89
+ To start with a clean slate and remove all installed gems:
56
90
 
57
91
  ```bash
58
92
  make clean_gems
59
93
  ```
60
94
 
95
+ > [!TIP]
96
+ > This is especially useful before testing across different Ruby versions or preparing for a release.
97
+
61
98
  ## Tool chain
62
99
 
63
100
  TODO: document installation of tool chain.
@@ -66,20 +103,24 @@ Build system uses GNU Make.
66
103
 
67
104
  ### Environment
68
105
 
69
- A few macros/envvars control some aspects:
106
+ A few macros/env vars control some aspects:
70
107
 
71
- | macro | description |
108
+ | `make` macro or Env var | Description |
72
109
  |-----------------------------|--------------------------------------|
73
110
  | `ASPERA_CLI_TEST_CONF_FILE` | Path to configuration file with secrets for tests |
74
111
  | `ASPERA_CLI_TEST_MACOS` | Set to `true` if local HSTS running on macOS |
75
112
  | `ASPERA_CLI_TEST_PRIVATE` | Path to private folder |
113
+ | `ASPERA_CLI_DOC_CHECK_LINKS`| Check links still exist during doc generation. |
114
+ | `ASPERA_CLI_DOC_DEBUG` | Enable debug in doc generation. |
76
115
  | `ENABLE_COVERAGE` | Tests with coverage analysis if set. |
77
116
  | `SIGNING_KEY` | Path to signing key to build Gem. |
78
117
  | `GEM_VERSION` | Gem version to build container |
79
118
 
80
119
  Those macros can be set either in an env var, or on the `make` command line.
81
120
 
82
- > **Note:** vars `ASPERA_CLI_TEST_` are typically set in user's shell profile.
121
+ > [!NOTE]
122
+ > Env vars `ASPERA_CLI_TEST_*` are typically set in user's shell profile for development.
123
+ > Others are more for "one shot" use.
83
124
 
84
125
  To use the CLI directly from the development environment, add this to your shell profile:
85
126
 
@@ -108,6 +149,10 @@ tlmgr update --self
108
149
  tlmgr install fvextra selnolig lualatex-math
109
150
  ```
110
151
 
152
+ To check URL during doc generation, set env var: `ASPERA_CLI_DOC_CHECK_LINKS=1`.
153
+
154
+ To debug doc generation, set env var: `ASPERA_CLI_DOC_DEBUG=debug`.
155
+
111
156
  ## Running Tests
112
157
 
113
158
  First, a testing configuration file must be created.
@@ -120,13 +165,13 @@ cp docs/test_env.conf ~/some_secure_folder/.
120
165
 
121
166
  Fill `~/some_secure_folder/test_env.conf` with system URLs and credentials for tests.
122
167
 
123
- Then tell where this file is located:
168
+ Then, tell where this file is located:
124
169
 
125
170
  ```bash
126
171
  export ASPERA_CLI_TEST_CONF_FILE=~/some_secure_folder/test_env.conf
127
172
  ```
128
173
 
129
- This project uses a `Makefile` for tests:
174
+ This project uses a `Makefile` for tests, in main folder:
130
175
 
131
176
  ```bash
132
177
  make test
@@ -134,6 +179,18 @@ make test
134
179
 
135
180
  When new commands are added to the CLI, new tests shall be added to the test suite in `tests/Makefile`.
136
181
 
182
+ One can also go to the `tests` folder:
183
+
184
+ ```bash
185
+ cd tests
186
+ make
187
+ make full
188
+ make list
189
+ make torc
190
+ make skip_torc
191
+ make skip T=orch_wizard
192
+ ```
193
+
137
194
  ### Special tests
138
195
 
139
196
  Some gems are optional: `rmagick` and `grpc`, as they require compilation of native code which may cause problems.