aspera-cli 4.11.0 → 4.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- checksums.yaml.gz.sig +0 -0
- data/BUGS.md +0 -1
- data/CHANGELOG.md +71 -52
- data/CONTRIBUTING.md +31 -6
- data/README.md +404 -259
- data/bin/asession +2 -2
- data/docs/test_env.conf +1 -0
- data/examples/aoc.rb +2 -2
- data/examples/dascli +11 -11
- data/examples/faspex4.rb +7 -7
- data/examples/node.rb +1 -1
- data/examples/proxy.pac +2 -2
- data/examples/server.rb +3 -3
- data/lib/aspera/aoc.rb +105 -40
- data/lib/aspera/cli/extended_value.rb +4 -4
- data/lib/aspera/cli/{formater.rb → formatter.rb} +7 -7
- data/lib/aspera/cli/listener/progress.rb +1 -1
- data/lib/aspera/cli/listener/progress_multi.rb +2 -2
- data/lib/aspera/cli/main.rb +18 -18
- data/lib/aspera/cli/manager.rb +5 -5
- data/lib/aspera/cli/plugin.rb +23 -20
- data/lib/aspera/cli/plugins/aoc.rb +75 -112
- data/lib/aspera/cli/plugins/ats.rb +6 -6
- data/lib/aspera/cli/plugins/config.rb +84 -83
- data/lib/aspera/cli/plugins/cos.rb +1 -1
- data/lib/aspera/cli/plugins/faspex.rb +38 -38
- data/lib/aspera/cli/plugins/faspex5.rb +187 -43
- data/lib/aspera/cli/plugins/node.rb +30 -37
- data/lib/aspera/cli/plugins/orchestrator.rb +7 -4
- data/lib/aspera/cli/plugins/preview.rb +10 -9
- data/lib/aspera/cli/plugins/server.rb +1 -1
- data/lib/aspera/cli/plugins/shares.rb +67 -43
- data/lib/aspera/cli/transfer_agent.rb +16 -16
- data/lib/aspera/cli/version.rb +2 -1
- data/lib/aspera/command_line_builder.rb +70 -66
- data/lib/aspera/cos_node.rb +9 -9
- data/lib/aspera/fasp/agent_base.rb +3 -1
- data/lib/aspera/fasp/agent_connect.rb +23 -23
- data/lib/aspera/fasp/agent_direct.rb +13 -14
- data/lib/aspera/fasp/agent_httpgw.rb +20 -19
- data/lib/aspera/fasp/agent_node.rb +13 -15
- data/lib/aspera/fasp/agent_trsdk.rb +1 -1
- data/lib/aspera/fasp/installation.rb +5 -5
- data/lib/aspera/fasp/listener.rb +1 -1
- data/lib/aspera/fasp/parameters.rb +49 -41
- data/lib/aspera/fasp/parameters.yaml +311 -212
- data/lib/aspera/fasp/resume_policy.rb +2 -2
- data/lib/aspera/fasp/transfer_spec.rb +0 -13
- data/lib/aspera/faspex_gw.rb +80 -161
- data/lib/aspera/faspex_postproc.rb +77 -0
- data/lib/aspera/log.rb +7 -7
- data/lib/aspera/nagios.rb +6 -6
- data/lib/aspera/node.rb +24 -19
- data/lib/aspera/oauth.rb +50 -47
- data/lib/aspera/proxy_auto_config.js +22 -22
- data/lib/aspera/proxy_auto_config.rb +3 -3
- data/lib/aspera/rest.rb +12 -10
- data/lib/aspera/rest_error_analyzer.rb +5 -5
- data/lib/aspera/secret_hider.rb +4 -3
- data/lib/aspera/ssh.rb +4 -4
- data/lib/aspera/sync.rb +37 -36
- data/lib/aspera/web_auth.rb +7 -59
- data/lib/aspera/web_server_simple.rb +76 -0
- data.tar.gz.sig +0 -0
- metadata +6 -4
- metadata.gz.sig +0 -0
data/README.md
CHANGED
@@ -1,11 +1,13 @@
|
|
1
1
|
# Command Line Interface for IBM Aspera products
|
2
2
|
<!-- markdownlint-disable MD033 MD003 MD053 -->
|
3
|
-
|
3
|
+
<!-- cSpell:ignore devkit zcvf zxvf noded secondfile filesize sedemo eudemo webmail csum eascp loglevel cronfile magick keepalive inotify eastus bluemix trev sshfp struct genkey passout ibmaspera unpermitted -->
|
4
|
+
<!-- cSpell:ignoreRegExp /-P[a-z]+/g -->
|
5
|
+
<!-- cSpell:ignoreRegExp /my[a-z]+/g -->
|
4
6
|
[comment1]: # (Do not edit this README.md, edit docs/README.erb.md, for details, read docs/README.md)
|
5
7
|
|
6
8
|
##
|
7
9
|
|
8
|
-
Version : 4.
|
10
|
+
Version : 4.12.0
|
9
11
|
|
10
12
|
Laurent/2016-2023
|
11
13
|
|
@@ -64,10 +66,20 @@ For scripting and ad'hoc command line operations, `ascli` is perfect.
|
|
64
66
|
|
65
67
|
## Notations, Shell, Examples
|
66
68
|
|
67
|
-
|
69
|
+
Command line operations examples are shown using a shell such: `bash` or `zsh`.
|
68
70
|
|
69
71
|
Command line parameters in examples beginning with `my_`, like `my_param_value` are user-provided value and not fixed value commands.
|
70
72
|
|
73
|
+
`ascli` is an API **Client** toward the remote Aspera application **Server** (Faspex, HSTS, etc...)
|
74
|
+
|
75
|
+
Some commands will start an Aspera-based transfer (e.g. `upload`).
|
76
|
+
The transfer is not directly implemented in `ascli`, rather `ascli` uses an external Aspera Client called **[Transfer Agents](#agents)**.
|
77
|
+
|
78
|
+
> **Note:** The transfer agent is a client for the remote Transfer Server (HSTS).
|
79
|
+
The transfer agent may be local or remote...
|
80
|
+
For example a remote Aspera Server may be used as a transfer agent (using node API).
|
81
|
+
i.e. using option `--transfer=node`
|
82
|
+
|
71
83
|
## Quick Start
|
72
84
|
|
73
85
|
This section guides you from installation, first use and advanced use.
|
@@ -81,7 +93,7 @@ ascli --version
|
|
81
93
|
```
|
82
94
|
|
83
95
|
```bash
|
84
|
-
4.
|
96
|
+
4.12.0
|
85
97
|
```
|
86
98
|
|
87
99
|
### First use
|
@@ -92,6 +104,8 @@ If you want to test with Aspera on Cloud, jump to section: [Wizard](#aocwizard)
|
|
92
104
|
|
93
105
|
To test with Aspera demo transfer server, setup the environment and then test:
|
94
106
|
|
107
|
+
<!-- spellchecker: disable -->
|
108
|
+
|
95
109
|
```bash
|
96
110
|
ascli config initdemo
|
97
111
|
```
|
@@ -111,6 +125,8 @@ ascli server browse /
|
|
111
125
|
:............:...........:......:........:...........................:.......................:
|
112
126
|
```
|
113
127
|
|
128
|
+
<!-- spellchecker: enable -->
|
129
|
+
|
114
130
|
If you want to use `ascli` with another server, and in order to make further calls more convenient, it is advised to define a [option preset](#lprt) for the server's authentication options. The following example will:
|
115
131
|
|
116
132
|
- create a [option preset](#lprt)
|
@@ -119,7 +135,7 @@ If you want to use `ascli` with another server, and in order to make further cal
|
|
119
135
|
- download a file
|
120
136
|
|
121
137
|
```bash
|
122
|
-
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
138
|
+
ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=my_password_here
|
123
139
|
```
|
124
140
|
|
125
141
|
```output
|
@@ -142,15 +158,10 @@ ascli server browse /aspera-test-dir-large
|
|
142
158
|
:............:...........:......:..............:...........................:............................:
|
143
159
|
: zmode : zuid : zgid : size : mtime : name :
|
144
160
|
:............:...........:......:..............:...........................:............................:
|
145
|
-
: -rw-rw-rw- : asperaweb : fasp : 10133504 : 2018-05-04 14:16:24 +0200 : ctl_female_2.fastq.partial :
|
146
161
|
: -rw-r--r-- : asperaweb : fasp : 209715200 : 2014-04-10 19:49:27 +0200 : 200MB :
|
147
162
|
: -rw-r--r-- : asperaweb : fasp : 524288000 : 2014-04-10 19:44:15 +0200 : 500MB :
|
148
163
|
: -rw-r--r-- : asperaweb : fasp : 5368709120 : 2014-04-10 19:45:52 +0200 : 5GB :
|
149
164
|
: -rw-r--r-- : asperaweb : fasp : 500000000000 : 2017-06-14 20:09:57 +0200 : 500GB :
|
150
|
-
: -rw-rw-rw- : asperaweb : fasp : 13606912 : 2018-05-04 14:20:21 +0200 : ctl_male_2.fastq.partial :
|
151
|
-
: -rw-rw-rw- : asperaweb : fasp : 76 : 2018-05-04 14:13:18 +0200 : ctl_female_2.fastq.haspx :
|
152
|
-
: -rw-rw-rw- : asperaweb : fasp : 647348 : 2018-05-04 14:26:39 +0200 : ctl_female_2.gz :
|
153
|
-
: -rw-rw-rw- : asperaweb : fasp : 74 : 2018-05-04 14:16:00 +0200 : ctl_male_2.fastq.haspx :
|
154
165
|
: -rw-r--r-- : asperaweb : fasp : 1048576000 : 2014-04-10 19:49:23 +0200 : 1GB :
|
155
166
|
: -rw-r--r-- : asperaweb : fasp : 104857600 : 2014-04-10 19:49:29 +0200 : 100MB :
|
156
167
|
: -rw-r--r-- : asperaweb : fasp : 10737418240 : 2014-04-10 19:49:04 +0200 : 10GB :
|
@@ -242,7 +253,7 @@ ascli -v
|
|
242
253
|
```
|
243
254
|
|
244
255
|
```text
|
245
|
-
4.
|
256
|
+
4.12.0
|
246
257
|
```
|
247
258
|
|
248
259
|
In order to keep persistency of configuration on the host,
|
@@ -290,7 +301,7 @@ A convenience sample script is also provided: download the script [`dascli`](../
|
|
290
301
|
|
291
302
|
> **Note:** If you have installed `ascli`, the script `dascli` can also be found: `cp $(ascli conf gem path)/../examples/dascli ascli`
|
292
303
|
|
293
|
-
Some environment variables can be set for this script to adapt its
|
304
|
+
Some environment variables can be set for this script to adapt its behavior:
|
294
305
|
|
295
306
|
| env var | description | default | example |
|
296
307
|
|--------------|------------------------------------|--------------------------|--------------------------|
|
@@ -652,7 +663,7 @@ Using `ascli` with plugin `server` for command line gives advantages over `ascp`
|
|
652
663
|
- choice of transfer agents
|
653
664
|
- integrated support of multi-session
|
654
665
|
|
655
|
-
Moreover all `ascp` options are supported either through transfer spec parameters and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`
|
666
|
+
Moreover all `ascp` options are supported either through transfer spec parameters and with the possibility to provide `ascp` arguments directly when the `direct` agent is used (`ascp_args`).
|
656
667
|
|
657
668
|
### <a id="parsing"></a>Command line parsing, Special Characters
|
658
669
|
|
@@ -702,13 +713,13 @@ ERROR: Argument: unprocessed values: ["2", "3"]
|
|
702
713
|
> **Note:** It gets its value after shell command line parsing and `ascli` extended value parsing.
|
703
714
|
|
704
715
|
In the following examples (using a POSIX shell, such as `bash`), several sample commands are provided when equivalent.
|
705
|
-
For all example, most of special character handling is not specific to `ascli`: It
|
716
|
+
For all example, most of special character handling is not specific to `ascli`: It depends on the underlying syntax: shell , JSON, etc...
|
706
717
|
Depending on the case, a different `format` is used to display the actual value.
|
707
718
|
|
708
719
|
For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented.
|
709
720
|
|
710
721
|
Double quotes are processed by the shell to create a single string argument.
|
711
|
-
For POSIX shells, single quotes can also be used in this case, or
|
722
|
+
For POSIX shells, single quotes can also be used in this case, or protect the special character ` ` (space) with a backslash. <!-- markdownlint-disable-line -->
|
712
723
|
|
713
724
|
```bash
|
714
725
|
ascli conf echo "Hello World" --format=text
|
@@ -798,9 +809,9 @@ ascli conf echo @ruby:"{'title'=>%q{Test \" ' & \\\\}}" --format=json
|
|
798
809
|
{"title":"Test \" ' & \\"}
|
799
810
|
```
|
800
811
|
|
801
|
-
#### Reading special characters
|
812
|
+
#### Reading special characters interactively
|
802
813
|
|
803
|
-
If `ascli` is used
|
814
|
+
If `ascli` is used interactively (a user typing on terminal), it is easy to require the user to type values:
|
804
815
|
|
805
816
|
```bash
|
806
817
|
ascli conf echo @ruby:"{'title'=>gets.chomp}" --format=json
|
@@ -956,7 +967,7 @@ The style of output can be set using the `format` parameter, supporting:
|
|
956
967
|
|
957
968
|
Table output can be filtered using the `select` parameter. Example:
|
958
969
|
|
959
|
-
```
|
970
|
+
```bash
|
960
971
|
ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'
|
961
972
|
```
|
962
973
|
|
@@ -1009,7 +1020,9 @@ The extended value syntax is:
|
|
1009
1020
|
<0 or more decoders><0 or 1 reader><nothing or some text value>
|
1010
1021
|
```
|
1011
1022
|
|
1012
|
-
The difference between reader and decoder is order and
|
1023
|
+
The difference between reader and decoder is order and cardinality.
|
1024
|
+
Both act like a function of value on right hand side.
|
1025
|
+
Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
|
1013
1026
|
|
1014
1027
|
The following "readers" are supported (returns value in []):
|
1015
1028
|
|
@@ -1019,13 +1032,13 @@ The following "readers" are supported (returns value in []):
|
|
1019
1032
|
- @path:PATH : [String] performs path expansion (prefix `~/` is replaced with the users home folder), e.g. `--config-file=@path:~/sample_config.yml`
|
1020
1033
|
- @env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
|
1021
1034
|
- @stdin: : [String] read from stdin (no value on right)
|
1022
|
-
- @preset:NAME : [Hash] get whole option preset value by name.
|
1035
|
+
- @preset:NAME : [Hash] get whole option preset value by name. Sub-values can also be used using `.` as separator. e.g. `foo.bar` is `conf[foo][bar]`
|
1023
1036
|
|
1024
1037
|
In addition it is possible to decode a value, using one or multiple decoders :
|
1025
1038
|
|
1026
1039
|
- @base64: [String] decode a base64 encoded string
|
1027
1040
|
- @json: [any] decode JSON values (convenient to provide complex structures)
|
1028
|
-
- @zlib: [String]
|
1041
|
+
- @zlib: [String] un-compress data
|
1029
1042
|
- @ruby: [any] execute ruby code
|
1030
1043
|
- @csvt: [Array] decode a titled CSV value
|
1031
1044
|
- @lines: [Array] split a string in multiple lines and return an array
|
@@ -1040,10 +1053,10 @@ Example: read the content of the specified file, then, base64 decode, then unzip
|
|
1040
1053
|
ascli config echo @zlib:@base64:@file:myfile.dat
|
1041
1054
|
```
|
1042
1055
|
|
1043
|
-
Example:
|
1056
|
+
Example: Create a value as a hash, with one key and the value is read from a file:
|
1044
1057
|
|
1045
1058
|
```bash
|
1046
|
-
ascli config echo @ruby:'{"token_verification_key"=>File.read("
|
1059
|
+
ascli config echo @ruby:'{"token_verification_key"=>File.read("mykey.txt")}'
|
1047
1060
|
```
|
1048
1061
|
|
1049
1062
|
Example: read a csv file and create a list of hash for bulk provisioning:
|
@@ -1073,7 +1086,7 @@ ascli config echo @csvt:@file:test.csv
|
|
1073
1086
|
|
1074
1087
|
Example: create a hash and include values from preset named "config" of config file in this hash
|
1075
1088
|
|
1076
|
-
```
|
1089
|
+
```bash
|
1077
1090
|
ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'
|
1078
1091
|
```
|
1079
1092
|
|
@@ -1123,7 +1136,7 @@ ascli config folder
|
|
1123
1136
|
C:\Users\Kenji\.aspera\ascli
|
1124
1137
|
```
|
1125
1138
|
|
1126
|
-
When OAuth is used (AoC, Faspex4
|
1139
|
+
When OAuth is used (AoC, Faspex4 api v4, Faspex5) `ascli` keeps a cache of generated bearer tokens in `[config folder]/persist_store` by default.
|
1127
1140
|
Option `cache_tokens` (**yes**/no) allows to control if Oauth tokens are cached on file system, or generated for each request.
|
1128
1141
|
The command `config flush_tokens` deletes all existing tokens.
|
1129
1142
|
Tokens are kept on disk for a maximum of 30 minutes (`TOKEN_CACHE_EXPIRY_SEC`) and garbage collected after that.
|
@@ -1157,7 +1170,7 @@ ascli config preset set|delete|show|initialize|update <option preset>
|
|
1157
1170
|
The command `update` allows the easy creation of [option preset](#lprt) by simply providing the options in their command line format, e.g. :
|
1158
1171
|
|
1159
1172
|
```bash
|
1160
|
-
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=
|
1173
|
+
ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=my_password_here --ts=@json:'{"precalculate_job_size":true}'
|
1161
1174
|
```
|
1162
1175
|
|
1163
1176
|
- This creates a [option preset](#lprt) `demo_server` with all provided options.
|
@@ -1165,13 +1178,13 @@ ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --u
|
|
1165
1178
|
The command `set` allows setting individual options in a [option preset](#lprt).
|
1166
1179
|
|
1167
1180
|
```bash
|
1168
|
-
ascli config preset set demo_server password
|
1181
|
+
ascli config preset set demo_server password my_password_here
|
1169
1182
|
```
|
1170
1183
|
|
1171
1184
|
The command `initialize`, like `update` allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a [*Structured Value*](#native).
|
1172
1185
|
|
1173
|
-
```
|
1174
|
-
ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"
|
1186
|
+
```bash
|
1187
|
+
ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"my_pass_here","ts":{"precalculate_job_size":true}}'
|
1175
1188
|
```
|
1176
1189
|
|
1177
1190
|
A full terminal based overview of the configuration can be displayed using:
|
@@ -1225,7 +1238,7 @@ ascli config preset set default _plugin_name_ _default_preset_for_plugin_
|
|
1225
1238
|
ascli config preset get default _plugin_name_
|
1226
1239
|
```
|
1227
1240
|
|
1228
|
-
```
|
1241
|
+
```json
|
1229
1242
|
"_default_preset_for_plugin_"
|
1230
1243
|
```
|
1231
1244
|
|
@@ -1295,7 +1308,7 @@ cli_default:
|
|
1295
1308
|
demo_server:
|
1296
1309
|
url: ssh://demo.asperasoft.com:33001
|
1297
1310
|
username: asperaweb
|
1298
|
-
password:
|
1311
|
+
password: my_password_here
|
1299
1312
|
```
|
1300
1313
|
|
1301
1314
|
We can see here:
|
@@ -1370,13 +1383,13 @@ ascli config preset unset cli_default interactive
|
|
1370
1383
|
Example: Define options using command line:
|
1371
1384
|
|
1372
1385
|
```bash
|
1373
|
-
ascli -N --url=_url_here_ --password=
|
1386
|
+
ascli -N --url=_url_here_ --password=my_password_here --username=_name_here_ node --show-config
|
1374
1387
|
```
|
1375
1388
|
|
1376
1389
|
Example: Define options using a hash:
|
1377
1390
|
|
1378
|
-
```
|
1379
|
-
ascli -N --preset=@json:'{"url":"_url_here_","password":"
|
1391
|
+
```bash
|
1392
|
+
ascli -N --preset=@json:'{"url":"_url_here_","password":"my_password_here","username":"_name_here_"}' node --show-config
|
1380
1393
|
```
|
1381
1394
|
|
1382
1395
|
#### Shares Examples
|
@@ -1386,7 +1399,7 @@ only username/password and url are required (either on command line, or from con
|
|
1386
1399
|
Those can usually be provided on the command line:
|
1387
1400
|
|
1388
1401
|
```bash
|
1389
|
-
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=
|
1402
|
+
ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=my_password_here
|
1390
1403
|
```
|
1391
1404
|
|
1392
1405
|
This can also be provisioned in a config file:
|
@@ -1396,19 +1409,19 @@ This can also be provisioned in a config file:
|
|
1396
1409
|
```bash
|
1397
1410
|
ascli config preset set shares06 url https://10.25.0.6
|
1398
1411
|
ascli config preset set shares06 username john
|
1399
|
-
ascli config preset set shares06 password
|
1412
|
+
ascli config preset set shares06 password my_password_here
|
1400
1413
|
```
|
1401
1414
|
|
1402
1415
|
This can also be done with one single command:
|
1403
1416
|
|
1404
|
-
```
|
1405
|
-
ascli config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"
|
1417
|
+
```bash
|
1418
|
+
ascli config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"my_password_here"}'
|
1406
1419
|
```
|
1407
1420
|
|
1408
1421
|
or
|
1409
1422
|
|
1410
1423
|
```bash
|
1411
|
-
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=
|
1424
|
+
ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=my_password_here
|
1412
1425
|
```
|
1413
1426
|
|
1414
1427
|
- Define this [option preset](#lprt) as the default [option preset](#lprt) for the specified plugin (`shares`)
|
@@ -1452,13 +1465,13 @@ export ASCLI_VAULT_PASSWORD
|
|
1452
1465
|
read -s ASCLI_VAULT_PASSWORD
|
1453
1466
|
```
|
1454
1467
|
|
1455
|
-
#### Vault: System
|
1468
|
+
#### Vault: System key chain
|
1456
1469
|
|
1457
1470
|
> **Note:** **macOS only**
|
1458
1471
|
|
1459
|
-
It is possible to manage secrets in macOS
|
1472
|
+
It is possible to manage secrets in macOS key chain (only read supported currently).
|
1460
1473
|
|
1461
|
-
```
|
1474
|
+
```bash
|
1462
1475
|
--vault=@json:'{"type":"system","name":"ascli"}'
|
1463
1476
|
```
|
1464
1477
|
|
@@ -1466,7 +1479,7 @@ It is possible to manage secrets in macOS keychain (only read supported currentl
|
|
1466
1479
|
|
1467
1480
|
It is possible to store and use secrets encrypted in a file.
|
1468
1481
|
|
1469
|
-
```
|
1482
|
+
```bash
|
1470
1483
|
--vault=@json:'{"type":"file","name":"vault.bin"}'
|
1471
1484
|
```
|
1472
1485
|
|
@@ -1484,7 +1497,7 @@ Then secrets can be manipulated using commands:
|
|
1484
1497
|
- `delete`
|
1485
1498
|
|
1486
1499
|
```bash
|
1487
|
-
ascli conf vault create mylabel @json:'{"password":"
|
1500
|
+
ascli conf vault create mylabel @json:'{"password":"my_password_here","description":"for this account"}'
|
1488
1501
|
```
|
1489
1502
|
|
1490
1503
|
#### <a id="config_finder"></a>Configuration Finder
|
@@ -1506,7 +1519,7 @@ For a more secure storage one can do:
|
|
1506
1519
|
|
1507
1520
|
```bash
|
1508
1521
|
`ascli` conf preset update myconf --url=... --username=... --password=@val:@vault:myconf.password
|
1509
|
-
`ascli` conf vault create myconf @json:'{"password":"
|
1522
|
+
`ascli` conf vault create myconf @json:'{"password":"my_password_here"}'
|
1510
1523
|
```
|
1511
1524
|
|
1512
1525
|
> **Note:** use `@val:` in front of `@vault:` so that the extended value is not evaluated.
|
@@ -1529,9 +1542,9 @@ Typically, in `$HOME/.ssh` or `$HOME/.aspera/ascli`:
|
|
1529
1542
|
PRIVKEYFILE=~/.aspera/ascli/my_private_key
|
1530
1543
|
```
|
1531
1544
|
|
1532
|
-
Several methods can be used to generate a key pair
|
1545
|
+
Several methods can be used to generate a key pair.
|
1533
1546
|
|
1534
|
-
|
1547
|
+
#### `ascli` for key generation
|
1535
1548
|
|
1536
1549
|
The generated key is of type RSA, by default: 4096 bit.
|
1537
1550
|
For convenience, the public key is also extracted with extension `.pub`.
|
@@ -1541,7 +1554,7 @@ The key is not passphrase protected.
|
|
1541
1554
|
ascli config genkey ${PRIVKEYFILE} 4096
|
1542
1555
|
```
|
1543
1556
|
|
1544
|
-
|
1557
|
+
#### `ssh-keygen`
|
1545
1558
|
|
1546
1559
|
Both private and public keys are generated, option `-N` is for passphrase.
|
1547
1560
|
|
@@ -1549,24 +1562,27 @@ Both private and public keys are generated, option `-N` is for passphrase.
|
|
1549
1562
|
ssh-keygen -t rsa -b 4096 -m PEM -N '' -f ${PRIVKEYFILE}
|
1550
1563
|
```
|
1551
1564
|
|
1552
|
-
|
1565
|
+
#### `openssl`
|
1553
1566
|
|
1554
1567
|
To generate a private key pair with a passphrase the following can be used on any system:
|
1555
|
-
|
1568
|
+
<!-- spellchecker: disable -->
|
1556
1569
|
```bash
|
1557
1570
|
openssl genrsa -passout pass:_passphrase_here_ -out ${PRIVKEYFILE}.protected 4096
|
1558
1571
|
openssl rsa -pubout -in ${PRIVKEYFILE} -out ${PRIVKEYFILE}.pub
|
1559
1572
|
```
|
1573
|
+
<!-- spellchecker: enable -->
|
1560
1574
|
|
1561
1575
|
`openssl` is sometimes compiled to support option `-nodes` (no DES, i.e. no passphrase, e.g. on macOS).
|
1562
1576
|
In that case, add option `-nodes` instead of `-passout pass:_passphrase_here_` to generate a key without passphrase.
|
1563
1577
|
|
1564
1578
|
If option `-nodes` is not available, the passphrase can be removed using this method:
|
1565
1579
|
|
1580
|
+
<!-- spellchecker: disable -->
|
1566
1581
|
```bash
|
1567
1582
|
openssl rsa -passin pass:_passphrase_here_ -in ${PRIVKEYFILE}.protected -out ${PRIVKEYFILE}
|
1568
1583
|
rm -f ${PRIVKEYFILE}.protected
|
1569
1584
|
```
|
1585
|
+
<!-- spellchecker: enable -->
|
1570
1586
|
|
1571
1587
|
To change (or add) the passphrase for a key do:
|
1572
1588
|
|
@@ -1610,9 +1626,9 @@ ascli conf plugin list
|
|
1610
1626
|
|
1611
1627
|
#### <a id="createownplugin"></a>Create your own plugin
|
1612
1628
|
|
1613
|
-
By default plugins are looked-up in folders
|
1629
|
+
By default plugins are looked-up in folders specified by (multi-value) option `plugin_folder`:
|
1614
1630
|
|
1615
|
-
```
|
1631
|
+
```bash
|
1616
1632
|
ascli --show-config --select=@json:'{"key":"plugin_folder"}'
|
1617
1633
|
```
|
1618
1634
|
|
@@ -1661,7 +1677,7 @@ Available loggers: `stdout`, `stderr`, `syslog`.
|
|
1661
1677
|
|
1662
1678
|
Available levels: `debug`, `info`, `warn`, `error`.
|
1663
1679
|
|
1664
|
-
> **Note:** When using the `direct` agent (`ascp`), additional transfer logs can be activated using `ascp`
|
1680
|
+
> **Note:** When using the `direct` agent (`ascp`), additional transfer logs can be activated using `ascp` options and `ascp_args`, see [`direct`](#agt_direct).
|
1665
1681
|
|
1666
1682
|
Examples:
|
1667
1683
|
|
@@ -1707,7 +1723,7 @@ Like any other option, those can be set either on command line, or in config fil
|
|
1707
1723
|
|
1708
1724
|
Example:
|
1709
1725
|
|
1710
|
-
```
|
1726
|
+
```bash
|
1711
1727
|
ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'
|
1712
1728
|
```
|
1713
1729
|
|
@@ -1739,7 +1755,7 @@ Refer to the following sections.
|
|
1739
1755
|
There are two possibilities to define an HTTP proxy to be used when Ruby HTTP is used.
|
1740
1756
|
|
1741
1757
|
The `http_proxy` environment variable (**lower case**, preferred) can be set to the URL of the proxy, e.g. `http://myproxy.org.net:3128`.
|
1742
|
-
Refer to [Ruby
|
1758
|
+
Refer to [Ruby find proxy](https://rubyapi.org/3.0/o/uri/generic#method-i-find_proxy).
|
1743
1759
|
|
1744
1760
|
> **Note:** Ruby expects a URL and `myproxy.org.net:3128` alone is **not** accepted.
|
1745
1761
|
|
@@ -1794,8 +1810,7 @@ ascli --proxy-credentials=@list::__username_here__:__password_here__ ...
|
|
1794
1810
|
|
1795
1811
|
### Proxy for Legacy Aspera HTTP/S Fallback
|
1796
1812
|
|
1797
|
-
To specify a proxy for legacy HTTP fallback, set the [*transfer-spec*](#transferspec) parameter: `EX_http_proxy_url
|
1798
|
-
(It is also possible to use `EX_ascp_args` and native options in `direct`)
|
1813
|
+
Only supported with the `direct` agent: To specify a proxy for legacy HTTP fallback, use `ascp` native option `-x` and `ascp_args`: `--transfer-info=@json:'{"ascp_args":["-x","url_here"]}'`. Alternatively, set the [*transfer-spec*](#transferspec) parameter: `EX_http_proxy_url`.
|
1799
1814
|
|
1800
1815
|
### FASP proxy (forward) for transfers
|
1801
1816
|
|
@@ -1803,7 +1818,8 @@ To specify a FASP proxy (forward), set the [*transfer-spec*](#transferspec) para
|
|
1803
1818
|
|
1804
1819
|
### <a id="client"></a>FASP configuration
|
1805
1820
|
|
1806
|
-
The `config` plugin also allows specification for the use of a local FASP client
|
1821
|
+
The `config` plugin also allows specification for the use of a local FASP **client**.
|
1822
|
+
It provides the following commands for `ascp` subcommand:
|
1807
1823
|
|
1808
1824
|
- `show` : shows the path of `ascp` used
|
1809
1825
|
- `use` : list,download connect client versions available on internet
|
@@ -1941,12 +1957,12 @@ Time: 00:00:02 =========================================================== 100%
|
|
1941
1957
|
Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg
|
1942
1958
|
```
|
1943
1959
|
|
1944
|
-
### <a id="agents"></a>Transfer Agents
|
1960
|
+
### <a id="agents"></a>Transfer Clients: Agents
|
1945
1961
|
|
1946
1962
|
Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`).
|
1947
1963
|
|
1948
1964
|
When a transfer needs to be started, a [*transfer-spec*](#transferspec) has been internally prepared.
|
1949
|
-
This [*transfer-spec*](#transferspec) will be executed by a transfer client, here called
|
1965
|
+
This [*transfer-spec*](#transferspec) will be executed by a transfer client, here called **Transfer Agent**.
|
1950
1966
|
|
1951
1967
|
There are currently 3 agents:
|
1952
1968
|
|
@@ -1975,6 +1991,7 @@ The `transfer_info` option accepts the following optional parameters to control
|
|
1975
1991
|
| Name | Type | Description |
|
1976
1992
|
|----------------------|-------|-------------|
|
1977
1993
|
| wss | Bool | Web Socket Session<br/>Enable use of web socket session in case it is available<br/>Default: true |
|
1994
|
+
| ascp_args | Array | Array of strings with native ascp arguments<br/>Use this instead of deprecated `EX_ascp_args`.<br/>Default: [] |
|
1978
1995
|
| spawn_timeout_sec | Float | Multi session<br/>Verification time that `ascp` is running<br/>Default: 3 |
|
1979
1996
|
| spawn_delay_sec | Float | Multi session<br/>Delay between startup of sessions<br/>Default: 2 |
|
1980
1997
|
| multi_incr_udp | Bool | Multi Session<br/>Increment UDP port on multi-session<br/>If true, each session will have a different UDP port starting at `fasp_port` (or default 33001)<br/>Else, each session will use `fasp_port` (or `ascp` default)<br/>Default: true |
|
@@ -2000,33 +2017,33 @@ ascli config ascp errors
|
|
2000
2017
|
|
2001
2018
|
Examples:
|
2002
2019
|
|
2003
|
-
```
|
2020
|
+
```bash
|
2004
2021
|
ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":20}}'
|
2005
2022
|
ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'
|
2006
2023
|
```
|
2007
2024
|
|
2008
2025
|
> **Note:** The `direct` agent supports additional `transfer_spec` parameters starting with `EX_` (extended).
|
2009
|
-
|
2026
|
+
But it is preferred to use the option `transfer_info` with parameter `ascp_args`.
|
2010
2027
|
|
2011
2028
|
This can be useful to activate logging using option `-L` of `ascp`.
|
2012
|
-
For example the option `--
|
2029
|
+
For example the option `--transfer-info=@json:'{"ascp_args":["-DDL-"]}'` will activate debug level 2 for `ascp` (`DD`), and display those logs on the terminal (`-`).
|
2013
2030
|
This is useful if the transfer fails.
|
2014
|
-
To store `ascp` logs in file `aspera-scp-transfer.log` in a folder, use `--
|
2031
|
+
To store `ascp` logs in file `aspera-scp-transfer.log` in a folder, use `--transfer-info=@json:'{"ascp_args":["-L","/path/to/folder"]}'`.
|
2015
2032
|
|
2016
|
-
> **Note:**
|
2033
|
+
> **Note:** When transfer agent [`direct`](#agt_direct) is used, the list of files to transfer is provided to `ascp` using either `--file-list` or `--file-pair-list` and a file list (or pair) file generated in a temporary folder. (unless `--file-list` or `--file-pair-list` is provided using `transfer_info` parameter `ascp_args`).
|
2017
2034
|
|
2018
2035
|
In addition to standard methods described in section [File List](#file_list), it is possible to specify the list of file using those additional methods:
|
2019
2036
|
|
2020
2037
|
- Using the pseudo [*transfer-spec*](#transferspec) parameter `EX_file_list`
|
2021
2038
|
|
2022
|
-
```
|
2023
|
-
--sources=@ts --ts=@json:'{"EX_file_list":"
|
2039
|
+
```bash
|
2040
|
+
--sources=@ts --ts=@json:'{"EX_file_list":"file_list.txt"}'
|
2024
2041
|
```
|
2025
2042
|
|
2026
|
-
- Using
|
2043
|
+
- Using option `transfer_info` parameter `ascp_args`
|
2027
2044
|
|
2028
|
-
```
|
2029
|
-
--sources=@ts --
|
2045
|
+
```bash
|
2046
|
+
--sources=@ts --transfer-info=@json:'{"ascp_args":["--file-list","myfilelist"]}'
|
2030
2047
|
```
|
2031
2048
|
|
2032
2049
|
> **Note:** File lists is shown here, there are also similar options for file pair lists.
|
@@ -2035,17 +2052,6 @@ In addition to standard methods described in section [File List](#file_list), it
|
|
2035
2052
|
>
|
2036
2053
|
> **Note:** Those methods have limitations: they apply **only** to the [`direct`](#agt_direct) transfer agent (i.e. local `ascp`) and not for Aspera on Cloud.
|
2037
2054
|
|
2038
|
-
In addition to special transfer spec parameter `EX_ascp_args`, it is possible to provide the same `ascp` options using option `ascp_opts` of `ascli`.
|
2039
|
-
This option expects an `Array`, which can be conveniently provided with extended syntax `@list:`
|
2040
|
-
|
2041
|
-
```bash
|
2042
|
-
ascli server download /aspera-test-dir-large/200MB --ascp-opts=@list:' -l 10m -k 3'
|
2043
|
-
```
|
2044
|
-
|
2045
|
-
> **Note:** Using `@list:`, the use of quotes and leading character "space" here is important: The `@list:` expects a single string which must not be parsed by the shell (so protected with quotes) and the leading space is the separator character.
|
2046
|
-
>
|
2047
|
-
> **Note:** The option `ascp_opts` are appended to `EX_ascp_args` if present.
|
2048
|
-
|
2049
2055
|
#### <a id="agt_connect"></a>IBM Aspera Connect Client GUI
|
2050
2056
|
|
2051
2057
|
By specifying option: `--transfer=connect`, `ascli` will start transfers using the locally installed Aspera Connect Client. There are no option for `transfer_info`.
|
@@ -2064,8 +2070,8 @@ Parameters provided in option `transfer_info` are:
|
|
2064
2070
|
| root_id | string | password or secret</br>Mandatory only for bearer token |
|
2065
2071
|
|
2066
2072
|
Like any other option, `transfer_info` can get its value from a pre-configured [option preset](#lprt) :
|
2067
|
-
`--transfer-info=@preset
|
2068
|
-
`--transfer-info=@json:'{"url":"https://...","username":"
|
2073
|
+
`--transfer-info=@preset:_name_here_` or be specified using the extended value syntax :
|
2074
|
+
`--transfer-info=@json:'{"url":"https://...","username":"_user_here_","password":"my_password_here"}'`
|
2069
2075
|
|
2070
2076
|
If `transfer_info` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default.
|
2071
2077
|
|
@@ -2077,15 +2083,15 @@ If it possible to send using a HTTP gateway, in case FASP is not allowed.
|
|
2077
2083
|
|
2078
2084
|
Parameters provided in option `transfer_info` are:
|
2079
2085
|
|
2080
|
-
| Name
|
2081
|
-
|
2082
|
-
| url
|
2083
|
-
| upload_bar_refresh_sec | float
|
2084
|
-
|
|
2086
|
+
| Name | Type | Description |
|
2087
|
+
|------------------------|--------|---------------------------------------|
|
2088
|
+
| url | string | URL of the HTTP GW</br>Mandatory |
|
2089
|
+
| upload_bar_refresh_sec | float | Refresh rate for upload progress bar |
|
2090
|
+
| upload_chunk_size | int | Size in bytes of chunks for upload |
|
2085
2091
|
|
2086
2092
|
Example:
|
2087
2093
|
|
2088
|
-
```
|
2094
|
+
```bash
|
2089
2095
|
ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'
|
2090
2096
|
```
|
2091
2097
|
|
@@ -2093,7 +2099,7 @@ ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"ur
|
|
2093
2099
|
|
2094
2100
|
#### <a id="agt_trsdk"></a>Transfer SDK
|
2095
2101
|
|
2096
|
-
Another possibility is to use the Transfer SDK daemon (asperatransferd).
|
2102
|
+
Another possibility is to use the Transfer SDK daemon (`asperatransferd`).
|
2097
2103
|
|
2098
2104
|
By default it will listen on local port `55002` on `127.0.0.1`.
|
2099
2105
|
|
@@ -2105,11 +2111,12 @@ gem install grpc
|
|
2105
2111
|
```
|
2106
2112
|
|
2107
2113
|
On Windows the compilation may fail for various reasons (3.1.1):
|
2108
|
-
|
2114
|
+
<!-- spellchecker: disable -->
|
2109
2115
|
- `cannot find -lx64-ucrt-ruby310`
|
2110
2116
|
→ copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` (remove the dll extension)
|
2111
2117
|
- `conflicting types for 'gettimeofday'`
|
2112
2118
|
→ edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)` ,i.e. change `struct timezone` to `void`
|
2119
|
+
<!-- spellchecker: enable -->
|
2113
2120
|
|
2114
2121
|
### <a id="transferspec"></a>Transfer Specification
|
2115
2122
|
|
@@ -2126,7 +2133,9 @@ is described in a [*transfer-spec*](#transferspec) (Transfer Specification), suc
|
|
2126
2133
|
|
2127
2134
|
If needed, it is possible to modify or add any of the supported [*transfer-spec*](#transferspec) parameter using the `ts` option. The `ts` option accepts a [Structured Value](#native) containing one or several [*transfer-spec*](#transferspec) parameters. Multiple `ts` options on command line are cumulative.
|
2128
2135
|
|
2129
|
-
It is possible to specify `ascp` options when the `transfer` option is set to [`direct`](#agt_direct) using
|
2136
|
+
It is possible to specify `ascp` options when the `transfer` option is set to [`direct`](#agt_direct) using `transfer_info` option parameter: `ascp_args`.
|
2137
|
+
Example: `--transfer-info=@json:'{"ascp_args":["-l","100m"]}'`.
|
2138
|
+
This is especially useful for `ascp` command line parameters not supported in the transfer spec.
|
2130
2139
|
|
2131
2140
|
The use of a [*transfer-spec*](#transferspec) instead of `ascp` parameters has the advantage of:
|
2132
2141
|
|
@@ -2148,7 +2157,7 @@ References:
|
|
2148
2157
|
|
2149
2158
|
Parameters can be displayed with commands:
|
2150
2159
|
|
2151
|
-
```
|
2160
|
+
```bash
|
2152
2161
|
ascli config ascp spec
|
2153
2162
|
ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c
|
2154
2163
|
```
|
@@ -2246,8 +2255,8 @@ Fields with EX_ prefix are extensions to transfer agent [`direct`](#agt_direct).
|
|
2246
2255
|
| title | string | | Y | Y | Title of the transfer |
|
2247
2256
|
| token | string | Y | Y | Y | Authorization token: Bearer, Basic or ATM (Also arg -W)<br/>(env:ASPERA_SCP_TOKEN) |
|
2248
2257
|
| use_ascp4 | bool | Y | Y | | specify version of protocol |
|
2249
|
-
| wss_enabled | bool | Y | Y | Y |
|
2250
|
-
| wss_port | int | Y | Y | Y | TCP port used for websocket service feed
|
2258
|
+
| wss_enabled | bool | Y | Y | Y | Server has Web Socket service enabled |
|
2259
|
+
| wss_port | int | Y | Y | Y | TCP port used for websocket service feed |
|
2251
2260
|
|
2252
2261
|
#### Destination folder for transfers
|
2253
2262
|
|
@@ -2312,7 +2321,7 @@ So, by default, the list of files to transfer will be simply specified on the co
|
|
2312
2321
|
|
2313
2322
|
- Using JSON array
|
2314
2323
|
|
2315
|
-
```
|
2324
|
+
```bash
|
2316
2325
|
--sources=@json:'["file1","file2"]'
|
2317
2326
|
```
|
2318
2327
|
|
@@ -2333,7 +2342,7 @@ Examples:
|
|
2333
2342
|
|
2334
2343
|
- Using transfer spec
|
2335
2344
|
|
2336
|
-
```
|
2345
|
+
```bash
|
2337
2346
|
--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
|
2338
2347
|
```
|
2339
2348
|
|
@@ -2360,7 +2369,7 @@ Multi session, i.e. starting a transfer of a file set using multiple sessions (o
|
|
2360
2369
|
|
2361
2370
|
- when agent=node :
|
2362
2371
|
|
2363
|
-
```
|
2372
|
+
```bash
|
2364
2373
|
--ts=@json:'{"multi_session":10,"multi_session_threshold":1}'
|
2365
2374
|
```
|
2366
2375
|
|
@@ -2368,7 +2377,7 @@ Multi-session is directly supported by the node daemon.
|
|
2368
2377
|
|
2369
2378
|
- when agent=direct :
|
2370
2379
|
|
2371
|
-
```
|
2380
|
+
```bash
|
2372
2381
|
--ts=@json:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}'
|
2373
2382
|
```
|
2374
2383
|
|
@@ -2380,7 +2389,7 @@ When multi-session is used, one separate UDP port is used per session (refer to
|
|
2380
2389
|
|
2381
2390
|
#### Content protection
|
2382
2391
|
|
2383
|
-
Also known as Client-side encryption at
|
2392
|
+
Also known as Client-side encryption at rest (CSEAR), content protection allows a client to send files to a server
|
2384
2393
|
which will store them encrypted (upload), and decrypt files as they are being downloaded from a server, both
|
2385
2394
|
using a passphrase, only known by users sharing files. Files stay encrypted on server side.
|
2386
2395
|
|
@@ -2391,39 +2400,33 @@ activating CSEAR consists in using transfer spec parameters:
|
|
2391
2400
|
|
2392
2401
|
Example: parameter to download a faspex package and decrypt on the fly
|
2393
2402
|
|
2394
|
-
```
|
2395
|
-
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"
|
2396
|
-
```
|
2397
|
-
|
2398
|
-
> **Note:** Up to version `ascli` 4.6.0, the following parameters should be used for agent `direct`:
|
2399
|
-
|
2400
|
-
```javascript
|
2401
|
-
--ts=@json:'{"EX_ascp_args":["--file-crypt=decrypt"],"EX_at_rest_password":"_secret_here_"}'
|
2403
|
+
```bash
|
2404
|
+
--ts=@json:'{"content_protection":"decrypt","content_protection_password":"my_password_here"}'
|
2402
2405
|
```
|
2403
2406
|
|
2404
2407
|
#### Transfer Spec Examples
|
2405
2408
|
|
2406
2409
|
- Change target rate
|
2407
2410
|
|
2408
|
-
```
|
2411
|
+
```bash
|
2409
2412
|
--ts=@json:'{"target_rate_kbps":500000}'
|
2410
2413
|
```
|
2411
2414
|
|
2412
2415
|
- Override the FASP SSH port to a specific TCP port:
|
2413
2416
|
|
2414
|
-
```
|
2417
|
+
```bash
|
2415
2418
|
--ts=@json:'{"ssh_port":33002}'
|
2416
2419
|
```
|
2417
2420
|
|
2418
2421
|
- Force http fallback mode:
|
2419
2422
|
|
2420
|
-
```
|
2423
|
+
```bash
|
2421
2424
|
--ts=@json:'{"http_fallback":"force"}'
|
2422
2425
|
```
|
2423
2426
|
|
2424
2427
|
- Activate progress when not activated by default on server
|
2425
2428
|
|
2426
|
-
```
|
2429
|
+
```bash
|
2427
2430
|
--ts=@json:'{"precalculate_job_size":true}'
|
2428
2431
|
```
|
2429
2432
|
|
@@ -2536,7 +2539,7 @@ where:
|
|
2536
2539
|
- `filename` is the name that will be assigned to the file on the destination
|
2537
2540
|
- `filesize` is the number of bytes that will be sent (in decimal).
|
2538
2541
|
|
2539
|
-
Note: characters `?` and `&` are shell special characters (wildcard and
|
2542
|
+
Note: characters `?` and `&` are shell special characters (wildcard and background), so `faux` file specification on command line should be protected (using quotes or `\`). If not, the shell may give error: `no matches found` or equivalent.
|
2540
2543
|
|
2541
2544
|
For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2<sup>20</sup>, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2<sup>60</sup>. Very large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`.
|
2542
2545
|
|
@@ -2571,7 +2574,7 @@ The sequence parameter is applied as follows:
|
|
2571
2574
|
|
2572
2575
|
- If `seq` is `sequential` then each file size is:
|
2573
2576
|
|
2574
|
-
- `size + ((
|
2577
|
+
- `size + ((file_index - 1) * inc)`
|
2575
2578
|
- Where first file is index 1
|
2576
2579
|
- So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, etc.
|
2577
2580
|
- As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less then 8*2<sup>60</sup>.
|
@@ -2605,7 +2608,7 @@ ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=seque
|
|
2605
2608
|
```text
|
2606
2609
|
ascli -h
|
2607
2610
|
NAME
|
2608
|
-
ascli -- a command line tool for Aspera Applications (v4.
|
2611
|
+
ascli -- a command line tool for Aspera Applications (v4.12.0)
|
2609
2612
|
|
2610
2613
|
SYNOPSIS
|
2611
2614
|
ascli COMMANDS [OPTIONS] [ARGS]
|
@@ -2653,7 +2656,7 @@ OPTIONS: global
|
|
2653
2656
|
-w, --warnings check for language warnings
|
2654
2657
|
--ui=ENUM method to start browser: text, [graphical]
|
2655
2658
|
--log-level=ENUM Log level: debug, info, [warn], error, fatal, unknown
|
2656
|
-
--logger=ENUM
|
2659
|
+
--logger=ENUM logging method: [stderr], stdout, syslog
|
2657
2660
|
--lock-port=VALUE prevent dual execution of a command, e.g. in cron
|
2658
2661
|
--http-options=VALUE options for http socket (extended value)
|
2659
2662
|
--insecure=ENUM do not validate HTTPS certificate: no, [yes]
|
@@ -2693,9 +2696,8 @@ OPTIONS:
|
|
2693
2696
|
--version-check-days=VALUE Period in days to check new version (zero to disable)
|
2694
2697
|
--plugin-folder=VALUE Folder where to find additional plugins
|
2695
2698
|
--ts=VALUE Override transfer spec values (Hash, e.g. use @json: prefix), current={"create_dir"=>true}
|
2696
|
-
--to-folder=VALUE Destination folder for
|
2697
|
-
--sources=VALUE How list of
|
2698
|
-
--ascp-opts=VALUE Options for ascp in its native format
|
2699
|
+
--to-folder=VALUE Destination folder for transferred files
|
2700
|
+
--sources=VALUE How list of transferred files is provided (@args,@ts,Array)
|
2699
2701
|
--src-type=ENUM Type of file list: list, pair
|
2700
2702
|
--transfer=ENUM Type of transfer agent: direct, node, connect, httpgw, trsdk
|
2701
2703
|
--transfer-info=VALUE Parameters for transfer agent
|
@@ -2708,6 +2710,7 @@ OPTIONS:
|
|
2708
2710
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2709
2711
|
--username=VALUE username to log in
|
2710
2712
|
--password=VALUE user's password
|
2713
|
+
--type=ENUM Type of user/group for operations: any, local, ldap, saml
|
2711
2714
|
|
2712
2715
|
|
2713
2716
|
COMMAND: node
|
@@ -2766,7 +2769,7 @@ OPTIONS:
|
|
2766
2769
|
|
2767
2770
|
|
2768
2771
|
COMMAND: faspex5
|
2769
|
-
SUBCOMMANDS: admin bearer_token health package user version
|
2772
|
+
SUBCOMMANDS: admin bearer_token gateway health package postprocessing shared_folders user version
|
2770
2773
|
OPTIONS:
|
2771
2774
|
--url=VALUE URL of application, e.g. https://org.asperafiles.com
|
2772
2775
|
--username=VALUE username to log in
|
@@ -2777,6 +2780,7 @@ OPTIONS:
|
|
2777
2780
|
--auth=ENUM OAuth type of authentication: boot, web, jwt
|
2778
2781
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
2779
2782
|
--passphrase=VALUE RSA private key passphrase
|
2783
|
+
--shared-folder=VALUE Shared folder source for package files
|
2780
2784
|
|
2781
2785
|
|
2782
2786
|
COMMAND: cos
|
@@ -2785,7 +2789,7 @@ OPTIONS:
|
|
2785
2789
|
--bucket=VALUE Bucket name
|
2786
2790
|
--endpoint=VALUE Storage endpoint url
|
2787
2791
|
--apikey=VALUE Storage API key
|
2788
|
-
--crn=VALUE
|
2792
|
+
--crn=VALUE Resource instance id
|
2789
2793
|
--service-credentials=VALUE IBM Cloud service credentials (Hash)
|
2790
2794
|
--region=VALUE Storage region
|
2791
2795
|
--identity=VALUE Authentication url (https://iam.cloud.ibm.com/identity)
|
@@ -2819,7 +2823,7 @@ OPTIONS:
|
|
2819
2823
|
--skip-folders=VALUE list of folder to skip
|
2820
2824
|
--case=VALUE basename of output for for test
|
2821
2825
|
--scan-path=VALUE subpath in folder id to start scan in (default=/)
|
2822
|
-
--scan-id=VALUE
|
2826
|
+
--scan-id=VALUE folder id in storage to start scan in, default is access key main folder id
|
2823
2827
|
--mimemagic=ENUM use Mime type detection of gem mimemagic: [no], yes
|
2824
2828
|
--overwrite=ENUM when to overwrite result file: always, never, [mtime]
|
2825
2829
|
--file-access=ENUM how to read and write files in repository: [local], remote
|
@@ -2855,8 +2859,8 @@ OPTIONS:
|
|
2855
2859
|
--password=VALUE user's password
|
2856
2860
|
--auth=ENUM OAuth type of authentication: web, jwt
|
2857
2861
|
--operation=ENUM client operation for transfers: push, pull
|
2858
|
-
--client-id=VALUE OAuth API client identifier
|
2859
|
-
--client-secret=VALUE OAuth API client
|
2862
|
+
--client-id=VALUE OAuth API client identifier
|
2863
|
+
--client-secret=VALUE OAuth API client secret
|
2860
2864
|
--redirect-uri=VALUE OAuth API client redirect URI
|
2861
2865
|
--private-key=VALUE OAuth JWT RSA private key PEM value (prefix file path with @file:)
|
2862
2866
|
--scope=VALUE OAuth scope for AoC API calls
|
@@ -3000,13 +3004,15 @@ If you did not use the wizard, you can also manually create a [option preset](#l
|
|
3000
3004
|
|
3001
3005
|
Lets create an [option preset](#lprt) called: `my_aoc_org` using `ask` interactive input (client info from previous step):
|
3002
3006
|
|
3007
|
+
<!-- spellchecker: disable -->
|
3003
3008
|
```bash
|
3004
3009
|
ascli config preset ask my_aoc_org url client_id client_secret
|
3005
3010
|
option: url> https://myorg.ibmaspera.com/
|
3006
|
-
option: client_id>
|
3007
|
-
option: client_secret>
|
3011
|
+
option: client_id> my_client_id_here
|
3012
|
+
option: client_secret> my_client_secret_here
|
3008
3013
|
updated: my_aoc_org
|
3009
3014
|
```
|
3015
|
+
<!-- spellchecker: enable -->
|
3010
3016
|
|
3011
3017
|
(This can also be done in one line using the command `config preset update my_aoc_org --url=...`)
|
3012
3018
|
|
@@ -3052,7 +3058,7 @@ ascli aoc admin res client list
|
|
3052
3058
|
:............:...............:
|
3053
3059
|
```
|
3054
3060
|
|
3055
|
-
```
|
3061
|
+
```bash
|
3056
3062
|
ascli aoc admin res client modify my_BJbQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
|
3057
3063
|
```
|
3058
3064
|
|
@@ -3187,19 +3193,19 @@ Examples:
|
|
3187
3193
|
|
3188
3194
|
- List users with `laurent` in name:
|
3189
3195
|
|
3190
|
-
```
|
3196
|
+
```bash
|
3191
3197
|
ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'
|
3192
3198
|
```
|
3193
3199
|
|
3194
3200
|
- List users who logged-in before a date:
|
3195
3201
|
|
3196
|
-
```
|
3202
|
+
```bash
|
3197
3203
|
ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
3198
3204
|
```
|
3199
3205
|
|
3200
3206
|
- List external users and sort in reverse alphabetical order using name:
|
3201
3207
|
|
3202
|
-
```
|
3208
|
+
```bash
|
3203
3209
|
ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'
|
3204
3210
|
```
|
3205
3211
|
|
@@ -3270,7 +3276,7 @@ The secret is provided using the `secret` option.
|
|
3270
3276
|
For example in a command like:
|
3271
3277
|
|
3272
3278
|
```bash
|
3273
|
-
ascli aoc admin res node --id=123 --secret="
|
3279
|
+
ascli aoc admin res node --id=123 --secret="my_secret_here" v3 info
|
3274
3280
|
```
|
3275
3281
|
|
3276
3282
|
It is also possible to store secrets in the [secret vault](#vault) and then automatically find the related secret using the [config finder](#config_finder).
|
@@ -3305,7 +3311,7 @@ The environment provided contains the following additional variable:
|
|
3305
3311
|
|
3306
3312
|
Example:
|
3307
3313
|
|
3308
|
-
```
|
3314
|
+
```bash
|
3309
3315
|
ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
|
3310
3316
|
--query=@json:'{"status":"completed","direction":"receive"}' \
|
3311
3317
|
--notif-to=active --notif-template=@file:mytemplate.erb
|
@@ -3330,7 +3336,7 @@ Refer to section "Examples" of [ATS](#ats) and substitute command `ats` with `ao
|
|
3330
3336
|
|
3331
3337
|
#### Example: Bulk creation of users
|
3332
3338
|
|
3333
|
-
```
|
3339
|
+
```bash
|
3334
3340
|
ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]'
|
3335
3341
|
```
|
3336
3342
|
|
@@ -3345,7 +3351,9 @@ ascli aoc admin res user create --bulk=yes @json:'[{"email":"dummyuser1@example.
|
|
3345
3351
|
|
3346
3352
|
#### Example: Find with filter and delete
|
3347
3353
|
|
3348
|
-
|
3354
|
+
<!-- spellchecker: disable -->
|
3355
|
+
|
3356
|
+
```bash
|
3349
3357
|
ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
|
3350
3358
|
```
|
3351
3359
|
|
@@ -3366,7 +3374,7 @@ thelist=$(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fiel
|
|
3366
3374
|
echo $thelist
|
3367
3375
|
```
|
3368
3376
|
|
3369
|
-
```
|
3377
|
+
```json
|
3370
3378
|
["113501","354061"]
|
3371
3379
|
```
|
3372
3380
|
|
@@ -3383,6 +3391,8 @@ ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete
|
|
3383
3391
|
:.......:.........:
|
3384
3392
|
```
|
3385
3393
|
|
3394
|
+
<!-- spellchecker: enable -->
|
3395
|
+
|
3386
3396
|
#### Example: <a id="deactuser"></a>Find deactivated users since more than 2 years
|
3387
3397
|
|
3388
3398
|
```ruby
|
@@ -3423,11 +3433,11 @@ ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q"
|
|
3423
3433
|
|
3424
3434
|
Examples of query (TODO: cleanup):
|
3425
3435
|
|
3426
|
-
```
|
3436
|
+
```json
|
3427
3437
|
{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}
|
3428
3438
|
```
|
3429
3439
|
|
3430
|
-
```
|
3440
|
+
```json
|
3431
3441
|
{"tag":"aspera.files.package_id=LA8OU3p8w"}
|
3432
3442
|
```
|
3433
3443
|
|
@@ -3439,7 +3449,7 @@ ascli aoc admin res node --secret=_secret_ v3 events
|
|
3439
3449
|
|
3440
3450
|
#### Example: Display members of a workspace
|
3441
3451
|
|
3442
|
-
```
|
3452
|
+
```bash
|
3443
3453
|
ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
|
3444
3454
|
```
|
3445
3455
|
|
@@ -3458,7 +3468,7 @@ ascli aoc admin res workspace_membership list --fields=member_type,manager,membe
|
|
3458
3468
|
|
3459
3469
|
Other query parameters:
|
3460
3470
|
|
3461
|
-
```
|
3471
|
+
```json
|
3462
3472
|
{"workspace_membership_through":true,"include_indirect":true}
|
3463
3473
|
```
|
3464
3474
|
|
@@ -3504,7 +3514,7 @@ ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_membe
|
|
3504
3514
|
|
3505
3515
|
#### Example: Get users who did not log since a date
|
3506
3516
|
|
3507
|
-
```
|
3517
|
+
```bash
|
3508
3518
|
ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'
|
3509
3519
|
```
|
3510
3520
|
|
@@ -3519,7 +3529,7 @@ ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:
|
|
3519
3529
|
|
3520
3530
|
#### Example: List "Limited" users
|
3521
3531
|
|
3522
|
-
```
|
3532
|
+
```bash
|
3523
3533
|
ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'
|
3524
3534
|
```
|
3525
3535
|
|
@@ -3599,13 +3609,15 @@ Then, create two shared folders located in two regions, in your files home, in a
|
|
3599
3609
|
|
3600
3610
|
Then, transfer between those:
|
3601
3611
|
|
3602
|
-
```
|
3612
|
+
```bash
|
3603
3613
|
ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
|
3604
3614
|
```
|
3605
3615
|
|
3606
3616
|
#### Example: create registration key to register a node
|
3607
3617
|
|
3608
|
-
|
3618
|
+
<!-- spellchecker: disable -->
|
3619
|
+
|
3620
|
+
```bash
|
3609
3621
|
ascli aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
|
3610
3622
|
```
|
3611
3623
|
|
@@ -3630,6 +3642,8 @@ ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res cli
|
|
3630
3642
|
+-----+---------+
|
3631
3643
|
```
|
3632
3644
|
|
3645
|
+
<!-- spellchecker: enable -->
|
3646
|
+
|
3633
3647
|
#### Example: Create a Node
|
3634
3648
|
|
3635
3649
|
AoC nodes as actually composed with two related entities:
|
@@ -3648,7 +3662,7 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
3648
3662
|
|
3649
3663
|
The creation options are the ones of ATS API, refer to the [section on ATS](#ats_params) for more details and examples.
|
3650
3664
|
|
3651
|
-
```
|
3665
|
+
```bash
|
3652
3666
|
ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'
|
3653
3667
|
```
|
3654
3668
|
|
@@ -3666,8 +3680,8 @@ So, for example, the creation of a node using ATS in IBM Cloud looks like (see o
|
|
3666
3680
|
|
3667
3681
|
Then use the returned address for the `url` key to actually create the AoC Node entity:
|
3668
3682
|
|
3669
|
-
```
|
3670
|
-
ascli aoc admin res node create @json:'{"name":"myname","access_key":"
|
3683
|
+
```bash
|
3684
|
+
ascli aoc admin res node create @json:'{"name":"myname","access_key":"myaccesskeyid","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'
|
3671
3685
|
```
|
3672
3686
|
|
3673
3687
|
Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_key create` on the private node itself.
|
@@ -3717,46 +3731,54 @@ Notes:
|
|
3717
3731
|
|
3718
3732
|
#### Example: Send a package with one file to two users, using their email
|
3719
3733
|
|
3720
|
-
```
|
3721
|
-
ascli aoc
|
3734
|
+
```bash
|
3735
|
+
ascli aoc packages send --value=@json:'{"name":"my title","note":"my note","recipients":["laurent.martin.aspera@fr.ibm.com","other@example.com"]}' my_file.dat
|
3722
3736
|
```
|
3723
3737
|
|
3724
3738
|
#### Example: Send a package to a shared inbox with metadata
|
3725
3739
|
|
3726
|
-
```
|
3727
|
-
ascli aoc
|
3740
|
+
```bash
|
3741
|
+
ascli aoc packages send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox With Meta"],"metadata":{"Project Id":"123","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' ~/Documents/Samples/200KB.1
|
3728
3742
|
```
|
3729
3743
|
|
3730
3744
|
It is also possible to use identifiers and API parameters:
|
3731
3745
|
|
3732
|
-
```
|
3733
|
-
ascli aoc
|
3746
|
+
```bash
|
3747
|
+
ascli aoc packages send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":[{"type":"dropbox","id":"12345"}],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1
|
3734
3748
|
```
|
3735
3749
|
|
3736
3750
|
#### Example: List packages in a given shared inbox
|
3737
3751
|
|
3738
3752
|
When user packages are listed, the following query is used:
|
3739
3753
|
|
3740
|
-
```
|
3754
|
+
```json
|
3741
3755
|
{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}
|
3742
3756
|
```
|
3743
3757
|
|
3744
|
-
To list packages in a shared inbox, the query has to be specified with withe the shared inbox by name or its identifier.
|
3758
|
+
To list packages in a shared inbox, the query has to be specified with withe the shared inbox by name or its identifier.
|
3759
|
+
Additional parameters can be specified, as supported by the API (to find out available filters, consult the API definition, or use the web interface in developer mode).
|
3760
|
+
The current workspace is added unless specified in the query.
|
3745
3761
|
|
3746
3762
|
Using shared inbox name:
|
3747
3763
|
|
3748
|
-
```
|
3764
|
+
```bash
|
3749
3765
|
ascli aoc packages list --query=@json:'{"dropbox_name":"My Shared Inbox","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
3750
3766
|
```
|
3751
3767
|
|
3752
3768
|
Using shared inbox identifier: first retrieve the id of the shared inbox, and then list packages with the appropriate filter.
|
3753
3769
|
|
3754
3770
|
```bash
|
3755
|
-
|
3771
|
+
shared_box_id=$(ascli aoc packages shared_inboxes show name 'My Shared Inbox' --format=csv --display=data --fields=id --transpose-single=no)
|
3772
|
+
```
|
3773
|
+
|
3774
|
+
```bash
|
3775
|
+
ascli aoc packages list --query=@json:'{"dropbox_id":"'$shared_box_id'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'
|
3756
3776
|
```
|
3757
3777
|
|
3758
|
-
|
3759
|
-
|
3778
|
+
#### Example: Receive all packages from a given shared inbox
|
3779
|
+
|
3780
|
+
```bash
|
3781
|
+
ascli aoc packages recv ALL --workspace=_workspace_ --once-only=yes --lock-port=12345 --query=@json:'{"dropbox_name":"_shared_inbox_name_","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}'
|
3760
3782
|
```
|
3761
3783
|
|
3762
3784
|
#### Example: Send a package with files from the Files app
|
@@ -3781,7 +3803,7 @@ ascli aoc files browse /src_folder
|
|
3781
3803
|
Let's send a package with the file `10M.dat` from subfolder /src_folder in a package:
|
3782
3804
|
|
3783
3805
|
```bash
|
3784
|
-
ascli aoc files node_info /src_folder --format=json --display=data | ascli aoc
|
3806
|
+
ascli aoc files node_info /src_folder --format=json --display=data | ascli aoc packages send --value=@json:'{"name":"test","recipients":["laurent.martin.aspera@fr.ibm.com"]}' 10M.dat --transfer=node --transfer-info=@json:@stdin:
|
3785
3807
|
```
|
3786
3808
|
|
3787
3809
|
#### <a id="aoccargo"></a>Receive new packages only (Cargo)
|
@@ -3917,19 +3939,19 @@ f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
|
|
3917
3939
|
- Find files older than 1 year on a given node and store in file list
|
3918
3940
|
|
3919
3941
|
```bash
|
3920
|
-
ascli aoc admin res node --name='my node name' --secret='
|
3942
|
+
ascli aoc admin res node --name='my node name' --secret='my_secret_here' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
|
3921
3943
|
```
|
3922
3944
|
|
3923
3945
|
- Delete the files, one by one
|
3924
3946
|
|
3925
3947
|
```bash
|
3926
|
-
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='
|
3948
|
+
cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='my_secret_here' v4 delete "$path" ;done
|
3927
3949
|
```
|
3928
3950
|
|
3929
3951
|
- Delete the files in bulk
|
3930
3952
|
|
3931
3953
|
```bash
|
3932
|
-
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='
|
3954
|
+
cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='my_secret_here' v3 delete @lines:@stdin:
|
3933
3955
|
```
|
3934
3956
|
|
3935
3957
|
### AoC sample commands
|
@@ -3937,10 +3959,10 @@ cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='
|
|
3937
3959
|
```bash
|
3938
3960
|
aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
|
3939
3961
|
aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
3940
|
-
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak1ibmcloud","secret":"
|
3962
|
+
aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak1ibmcloud","secret":"my_secret_here","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
3941
3963
|
aoc admin ats access_key delete ak1ibmcloud
|
3942
3964
|
aoc admin ats access_key list --fields=name,id
|
3943
|
-
aoc admin ats access_key node ak1ibmcloud --secret=
|
3965
|
+
aoc admin ats access_key node ak1ibmcloud --secret=my_secret_here browse /
|
3944
3966
|
aoc admin ats cluster clouds
|
3945
3967
|
aoc admin ats cluster list
|
3946
3968
|
aoc admin ats cluster show --cloud=aws --region=eu-west-1
|
@@ -4007,6 +4029,7 @@ aoc files upload --to-folder=/ testfile.bin --link=my_aoc_publink_folder
|
|
4007
4029
|
aoc files upload --to-folder=/testsrc testfile.bin
|
4008
4030
|
aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
|
4009
4031
|
aoc files v3 info
|
4032
|
+
aoc gateway --value=https://localhost:12345/aspera/faspex & jobs -p
|
4010
4033
|
aoc org --link=my_aoc_publink_recv_from_aocuser
|
4011
4034
|
aoc organization
|
4012
4035
|
aoc packages browse "my_package_id" /contents
|
@@ -4014,6 +4037,7 @@ aoc packages list
|
|
4014
4037
|
aoc packages list --query=@json:'{"dropbox_name":"my_aoc_shbx_name","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
|
4015
4038
|
aoc packages recv "my_package_id" --to-folder=.
|
4016
4039
|
aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345
|
4040
|
+
aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345 --query=@json:'{"dropbox_name":"my_aoc_shbx_name","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}'
|
4017
4041
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_external_user"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
|
4018
4042
|
aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_internal_user"],"note":"my note"}' testfile.bin
|
4019
4043
|
aoc packages send --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
|
@@ -4041,10 +4065,12 @@ ATS is usable either :
|
|
4041
4065
|
### IBM Cloud ATS : creation of api key
|
4042
4066
|
|
4043
4067
|
This section is about using ATS with an IBM cloud subscription.
|
4044
|
-
If you are using ATS as part of AoC, then authentication is
|
4068
|
+
If you are using ATS as part of AoC, then authentication is through AoC, not IBM Cloud.
|
4045
4069
|
|
4046
4070
|
First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:
|
4047
4071
|
|
4072
|
+
<!-- spellchecker: disable -->
|
4073
|
+
|
4048
4074
|
```bash
|
4049
4075
|
ibmcloud iam api-key-create mykeyname -d 'my sample key'
|
4050
4076
|
```
|
@@ -4058,7 +4084,7 @@ Please preserve the API key! It cannot be retrieved after it's created.
|
|
4058
4084
|
Name mykeyname
|
4059
4085
|
Description my sample key
|
4060
4086
|
Created At 2019-09-30T12:17+0000
|
4061
|
-
API Key
|
4087
|
+
API Key my_secret_api_key_here
|
4062
4088
|
Locked false
|
4063
4089
|
UUID ApiKey-05b8fadf-e7fe-4bc4-93a9-6fd348c5ab1f
|
4064
4090
|
```
|
@@ -4068,10 +4094,14 @@ References:
|
|
4068
4094
|
- [https://console.bluemix.net/docs/iam/userid_keys.html#userapikey](https://console.bluemix.net/docs/iam/userid_keys.html#userapikey)
|
4069
4095
|
- [https://ibm.ibmaspera.com/helpcenter/transfer-service](https://ibm.ibmaspera.com/helpcenter/transfer-service)
|
4070
4096
|
|
4097
|
+
<!-- spellchecker: enable -->
|
4098
|
+
|
4071
4099
|
Then, to register the key by default for the ats plugin, create a preset. Execute:
|
4072
4100
|
|
4101
|
+
<!-- spellchecker: disable -->
|
4102
|
+
|
4073
4103
|
```bash
|
4074
|
-
ascli config preset update my_ibm_ats --ibm-api-key=
|
4104
|
+
ascli config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here
|
4075
4105
|
```
|
4076
4106
|
|
4077
4107
|
```bash
|
@@ -4108,36 +4138,39 @@ ascli ats api_key create
|
|
4108
4138
|
ascli config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
|
4109
4139
|
```
|
4110
4140
|
|
4141
|
+
<!-- spellchecker: enable -->
|
4142
|
+
|
4111
4143
|
### <a id="ats_params"></a>ATS Access key creation parameters
|
4112
4144
|
|
4113
|
-
When creating an ATS access key, the option `params` must contain an extended value with the creation parameters.
|
4145
|
+
When creating an ATS access key, the option `params` must contain an extended value with the creation parameters.
|
4146
|
+
Those are directly the parameters expected by the [ATS API](https://developer.ibm.com/apis/catalog?search=%22Aspera%20ATS%20API%22).
|
4114
4147
|
|
4115
4148
|
### Misc. Examples
|
4116
4149
|
|
4117
4150
|
Example: create access key on IBM Cloud (softlayer):
|
4118
4151
|
|
4119
|
-
```
|
4120
|
-
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"
|
4152
|
+
```bash
|
4153
|
+
ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"my_secret_here","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'
|
4121
4154
|
```
|
4122
4155
|
|
4123
4156
|
Example: create access key on AWS:
|
4124
4157
|
|
4125
|
-
```
|
4126
|
-
ascli ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"
|
4158
|
+
```bash
|
4159
|
+
ascli ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"myaccesskey","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"_access_key_id_here_","secret_access_key":"my_secret_here"},"path":"/laurent"}}'
|
4127
4160
|
```
|
4128
4161
|
|
4129
4162
|
Example: create access key on Azure SAS:
|
4130
4163
|
|
4131
|
-
```
|
4132
|
-
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"
|
4164
|
+
```bash
|
4165
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"myaccesskey","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'
|
4133
4166
|
```
|
4134
4167
|
|
4135
4168
|
(Note that the blob name is mandatory after server address and before parameters. and that parameter sr=c is mandatory.)
|
4136
4169
|
|
4137
4170
|
Example: create access key on Azure:
|
4138
4171
|
|
4139
|
-
```
|
4140
|
-
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"
|
4172
|
+
```bash
|
4173
|
+
ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"myaccesskey","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'
|
4141
4174
|
```
|
4142
4175
|
|
4143
4176
|
delete all my access keys:
|
@@ -4151,14 +4184,14 @@ The parameters provided to ATS for access key creation are the ones of [ATS API]
|
|
4151
4184
|
### ATS sample commands
|
4152
4185
|
|
4153
4186
|
```bash
|
4154
|
-
ats access_key cluster ak2ibmcloud --secret=
|
4187
|
+
ats access_key cluster ak2ibmcloud --secret=my_secret_here
|
4155
4188
|
ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
|
4156
|
-
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak2ibmcloud","secret":"
|
4189
|
+
ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"ak2ibmcloud","secret":"my_secret_here","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
|
4157
4190
|
ats access_key delete ak2ibmcloud
|
4158
4191
|
ats access_key delete ak_aws
|
4159
4192
|
ats access_key entitlement ak2ibmcloud
|
4160
4193
|
ats access_key list --fields=name,id
|
4161
|
-
ats access_key node ak2ibmcloud browse / --secret=
|
4194
|
+
ats access_key node ak2ibmcloud browse / --secret=my_secret_here
|
4162
4195
|
ats access_key show ak2ibmcloud
|
4163
4196
|
ats api_key create
|
4164
4197
|
ats api_key instances
|
@@ -4209,8 +4242,8 @@ server upload --sources=@ts --ts=@json:'{"EX_file_list":"'"filelist.txt"'"}' --t
|
|
4209
4242
|
server upload --sources=@ts --ts=@json:'{"EX_file_pair_list":"'"filepairlist.txt"'"}'
|
4210
4243
|
server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
|
4211
4244
|
server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
|
4212
|
-
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email --
|
4213
|
-
server upload --src-type=pair testfile.bin folder_1/with_options --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"
|
4245
|
+
server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email --transfer-info=@json:'{"ascp_args":["-l","10m"]}'
|
4246
|
+
server upload --src-type=pair testfile.bin folder_1/with_options --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"my_secret_here","cookie":"biscuit","create_dir":true,"delete_before_transfer":false,"delete_source":false,"exclude_newer_than":1,"exclude_older_than":10000,"fasp_port":33001,"http_fallback":false,"multi_session":0,"overwrite":"diff+older","precalculate_job_size":true,"preserve_access_time":true,"preserve_creation_time":true,"rate_policy":"fair","resume_policy":"sparse_csum","symlink_policy":"follow"}'
|
4214
4247
|
server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
|
4215
4248
|
server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}' --progress=multi
|
4216
4249
|
```
|
@@ -4228,7 +4261,7 @@ If no SSH password or key is provided and a transfer token is provided in transf
|
|
4228
4261
|
Example:
|
4229
4262
|
|
4230
4263
|
```bash
|
4231
|
-
ascli server --url=ssh://_server_address_:33001 ... --ts=@json:'{"token":"Basic
|
4264
|
+
ascli server --url=ssh://_server_address_:33001 ... --ts=@json:'{"token":"Basic _token_here_"}'
|
4232
4265
|
```
|
4233
4266
|
|
4234
4267
|
> **Note:** If you need to use the Aspera public keys, then specify an empty token: `--ts=@json:'{"token":""}'` : Aspera public SSH keys will be used, but the protocol will ignore the empty token.
|
@@ -4245,9 +4278,16 @@ ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]'
|
|
4245
4278
|
```
|
4246
4279
|
|
4247
4280
|
For file operation command (browse, delete), the ruby SSH client library `Net::SSH` is used and provides several options settable using option `ssh_options`.
|
4248
|
-
For a list of SSH client options, refer to the ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html).
|
4249
4281
|
|
4250
|
-
|
4282
|
+
For a list of SSH client options, refer to the ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html#method-c-start).
|
4283
|
+
|
4284
|
+
Some of the 50 available SSH options:
|
4285
|
+
|
4286
|
+
- `verbose`
|
4287
|
+
- `use_agent`
|
4288
|
+
- `passphrase`
|
4289
|
+
|
4290
|
+
By default the SSH library will check if a local `ssh-agent` is running.
|
4251
4291
|
|
4252
4292
|
On Linux, if you get an error message such as:
|
4253
4293
|
|
@@ -4261,25 +4301,26 @@ or on Windows:
|
|
4261
4301
|
ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running
|
4262
4302
|
```
|
4263
4303
|
|
4264
|
-
This means that you don't have such an SSH agent running, then:
|
4304
|
+
This means that your environment suggessts to use an agent but you don't have such an SSH agent running, then:
|
4265
4305
|
|
4266
4306
|
- Check env var: `SSH_AGENT_SOCK`
|
4307
|
+
- Check your file: `$HOME/.ssh/config`
|
4267
4308
|
- Check if the SSH key is protected with a passphrase (then, use the `passphrase` SSH option)
|
4268
|
-
- [
|
4309
|
+
- [Check the Ruby SSH manual](https://www.rubydoc.info/github/net-ssh/net-ssh/Net%2FSSH.start)
|
4269
4310
|
- To disable the use of `ssh-agent`, use the option `ssh_options` like this:
|
4270
4311
|
|
4271
4312
|
```bash
|
4272
|
-
ascli server --ssh-options=@
|
4313
|
+
ascli server --ssh-options=@json:'{"use_agent": false}' ...
|
4273
4314
|
```
|
4274
4315
|
|
4275
|
-
This can also be set
|
4316
|
+
> **Note:** This can also be set using a preset.
|
4276
4317
|
|
4277
4318
|
### Other session channels for `server`
|
4278
4319
|
|
4279
|
-
URL schemes `local` and `https` are also supported
|
4320
|
+
URL schemes `local` and `https` are also supported (mainly for testing purpose).
|
4280
4321
|
(`--url=local:` , `--url=https://...`)
|
4281
4322
|
|
4282
|
-
- `local` will execute `ascmd` locally, instead of using
|
4323
|
+
- `local` will execute `ascmd` locally, instead of using an SSH connection.
|
4283
4324
|
- `https` will use Web Socket Session: This requires the use of a transfer token. For example a `Basic` token can be used.
|
4284
4325
|
|
4285
4326
|
As, most of the time, SSH is used, if an `http` scheme is provided without token, the plugin will fallback to SSH and port 33001.
|
@@ -4314,7 +4355,7 @@ For transfers, it is possible to control how transfer is authorized using option
|
|
4314
4355
|
- `hybrid` : same as `aspera`, but token is replaced with basic token like `basic`
|
4315
4356
|
- `basic` : transfer spec is created like this:
|
4316
4357
|
|
4317
|
-
```
|
4358
|
+
```json
|
4318
4359
|
{
|
4319
4360
|
"remote_host": "<address of node url>",
|
4320
4361
|
"remote_user": "xfer",
|
@@ -4344,8 +4385,8 @@ It is possible to start a FASPStream session using the node API:
|
|
4344
4385
|
|
4345
4386
|
Use the "node stream create" command, then arguments are provided as a [*transfer-spec*](#transferspec).
|
4346
4387
|
|
4347
|
-
```
|
4348
|
-
ascli node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"
|
4388
|
+
```bash
|
4389
|
+
ascli node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"my_pass_here"}' --preset=stream
|
4349
4390
|
```
|
4350
4391
|
|
4351
4392
|
### Watchfolder
|
@@ -4357,17 +4398,21 @@ Refer to [Aspera documentation](https://download.asperasoft.com/download/docs/en
|
|
4357
4398
|
- Start watchd and watchfolderd services running as a system user having access to files
|
4358
4399
|
- configure a watchfolder to define automated transfers
|
4359
4400
|
|
4360
|
-
|
4401
|
+
<!-- spellchecker: disable -->
|
4402
|
+
|
4403
|
+
```bash
|
4361
4404
|
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
|
4362
4405
|
ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
|
4363
4406
|
ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'
|
4364
4407
|
```
|
4365
4408
|
|
4409
|
+
<!-- spellchecker: enable -->
|
4410
|
+
|
4366
4411
|
### Out of Transfer File Validation
|
4367
4412
|
|
4368
4413
|
Follow the Aspera Transfer Server configuration to activate this feature.
|
4369
4414
|
|
4370
|
-
```
|
4415
|
+
```bash
|
4371
4416
|
ascli node central file list --validator=ascli --data=@json:'{"file_transfer_filter":{"max_result":1}}'
|
4372
4417
|
```
|
4373
4418
|
|
@@ -4379,7 +4424,7 @@ ascli node central file list --validator=ascli --data=@json:'{"file_transfer_fil
|
|
4379
4424
|
:..............:..............:............:......................................:
|
4380
4425
|
```
|
4381
4426
|
|
4382
|
-
```
|
4427
|
+
```bash
|
4383
4428
|
ascli node central file update --validator=ascli --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
|
4384
4429
|
```
|
4385
4430
|
|
@@ -4389,23 +4434,23 @@ updated
|
|
4389
4434
|
|
4390
4435
|
### Example: SHOD to ATS
|
4391
4436
|
|
4392
|
-
Scenario: Access to a
|
4437
|
+
Scenario: Access to a **Shares on Demand** (SHOD) server on AWS is provided by a partner.
|
4393
4438
|
We need to transfer files from this third party SHOD instance into our Azure BLOB storage.
|
4394
|
-
Simply create an
|
4395
|
-
Then create a configuration for the
|
4396
|
-
Create another configuration for the Azure ATS instance: in section "node", named
|
4439
|
+
Simply create an **Aspera Transfer Service** instance, which provides access to the node API.
|
4440
|
+
Then create a configuration for the **SHOD** instance in the configuration file: in section "shares", a configuration named: aws_shod.
|
4441
|
+
Create another configuration for the Azure ATS instance: in section "node", named azure_ats.
|
4397
4442
|
Then execute the following command:
|
4398
4443
|
|
4399
4444
|
```bash
|
4400
|
-
ascli node download /share/sourcefile --to-folder=/
|
4445
|
+
ascli node download /share/sourcefile --to-folder=/destination_folder --preset=aws_shod --transfer=node --transfer-info=@preset:azure_ats
|
4401
4446
|
```
|
4402
4447
|
|
4403
4448
|
This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.
|
4404
4449
|
|
4405
4450
|
### Create access key
|
4406
4451
|
|
4407
|
-
```
|
4408
|
-
ascli node access_key create --value=@json:'{"id":"
|
4452
|
+
```bash
|
4453
|
+
ascli node access_key create --value=@json:'{"id":"myaccesskey","secret":"my_secret_here","storage":{"type":"local","path":"/data/mydir"}}'
|
4409
4454
|
```
|
4410
4455
|
|
4411
4456
|
### Node sample commands
|
@@ -4464,7 +4509,7 @@ node sync state my_syncid
|
|
4464
4509
|
node sync stop my_syncid
|
4465
4510
|
node sync summary my_syncid
|
4466
4511
|
node transfer list --value=@json:'{"active_only":true}'
|
4467
|
-
node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"
|
4512
|
+
node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"my_node_pass_here"}'
|
4468
4513
|
node upload --username=my_aoc_ak_name --password=my_aoc_ak_secret testfile.bin --token-type=basic
|
4469
4514
|
node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}'
|
4470
4515
|
node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}' --token-type=hybrid
|
@@ -4482,7 +4527,7 @@ IBM Aspera's newer self-managed application.
|
|
4482
4527
|
|
4483
4528
|
### Faspex 5 JWT authentication
|
4484
4529
|
|
4485
|
-
This is the **
|
4530
|
+
This is the **recommended** method to use.
|
4486
4531
|
|
4487
4532
|
For `jwt`, create an API client in Faspex with JWT support:
|
4488
4533
|
|
@@ -4498,7 +4543,7 @@ Then use these options:
|
|
4498
4543
|
```text
|
4499
4544
|
--auth=jwt
|
4500
4545
|
--client-id=_client_id_here_
|
4501
|
-
--client-secret=
|
4546
|
+
--client-secret=my_secret_here
|
4502
4547
|
--username=_username_here_
|
4503
4548
|
--private-key=@file:.../path/to/key.pem
|
4504
4549
|
```
|
@@ -4520,7 +4565,7 @@ Then use options:
|
|
4520
4565
|
```text
|
4521
4566
|
--auth=web
|
4522
4567
|
--client-id=_client_id_here_
|
4523
|
-
--client-secret=
|
4568
|
+
--client-secret=my_secret_here
|
4524
4569
|
--redirect-uri=https://127.0.0.1:8888
|
4525
4570
|
```
|
4526
4571
|
|
@@ -4539,10 +4584,19 @@ Use this token as password and use `--auth=boot`.
|
|
4539
4584
|
ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=_token_here_
|
4540
4585
|
```
|
4541
4586
|
|
4587
|
+
### Faspex 5 packages
|
4588
|
+
|
4589
|
+
The `value` option provided to command `faspex5 package send` is the same as for the Faspex 5 API: `POST /packages`.
|
4590
|
+
|
4591
|
+
In addition, `ascli` adds some convenience: the field `recipients` is normally an Array of Hash, each with field `name` and optionally `recipient_type`, but it is also possible to provide an Array of String, with simply a recipient name.
|
4592
|
+
Then `ascli` will lookup existing contacts, and if a single match is found will use it, and set the `name` and `recipient_type` accordingly.
|
4593
|
+
|
4594
|
+
> **Note:** The lookup is case insensitive and on partial matches.
|
4595
|
+
|
4542
4596
|
### Faspex 5 sample commands
|
4543
4597
|
|
4544
4598
|
Most commands are directly REST API calls.
|
4545
|
-
Parameters to
|
4599
|
+
Parameters to commands are carried through option `value`, as extended value.
|
4546
4600
|
Usually using JSON format with prefix `@json:`.
|
4547
4601
|
|
4548
4602
|
> **Note:** The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**.
|
@@ -4559,40 +4613,114 @@ faspex5 admin res saml_configs list
|
|
4559
4613
|
faspex5 admin res shared_inboxes list
|
4560
4614
|
faspex5 admin res workgroups list
|
4561
4615
|
faspex5 bearer_token
|
4616
|
+
faspex5 gateway --value=https://localhost:12345/aspera/faspex &\
|
4562
4617
|
faspex5 health
|
4563
4618
|
faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
|
4564
4619
|
faspex5 package receive "my_package_id" --to-folder=. --ts=@json:'{"content_protection_password":"abc123_yo"}'
|
4565
|
-
faspex5 package send --value=@json:'{"title":"test title","recipients":[{"
|
4620
|
+
faspex5 package send --value=@json:'{"title":"test title","recipients":["my_shinbox"],"metadata":{"Options":"Opt1","TextInput":"example text"}}' testfile.bin
|
4621
|
+
faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"my_f5_user"}]}' testfile.bin --ts=@json:'{"content_protection_password":"my_passphrase_here"}'
|
4566
4622
|
faspex5 package show "my_package_id"
|
4623
|
+
faspex5 postprocessing --value=@json:'{"url":"https://localhost:8443/domain","processing":{"script_folder":"tests"},"certificate":{"key":"../local/k","cert":"../local/c","chain":"../local/ch"}}' &\
|
4567
4624
|
faspex5 user profile modify @json:'{"preference":{"connect_disabled":false}}'
|
4568
4625
|
faspex5 user profile show
|
4569
4626
|
```
|
4570
4627
|
|
4571
4628
|
Other examples:
|
4572
4629
|
|
4630
|
+
- Send a package with metadata
|
4631
|
+
|
4632
|
+
The interface is the one of the API (Refer to API documentation, or look at request in browser):
|
4633
|
+
|
4634
|
+
```bash
|
4635
|
+
ascli faspex5 package send --value=@json:'{"title":"test title","recipients":["ascli shared inbox"],"metadata":{"Confidential":"Yes","Drop menu":"Option 1"}}' 'faux:///test1?k1'
|
4636
|
+
```
|
4637
|
+
|
4638
|
+
Basically, add the field `metadata`, with one key per metadata and the value is directly the metadata value.
|
4639
|
+
|
4573
4640
|
- List all shared inboxes
|
4574
4641
|
|
4575
|
-
```
|
4642
|
+
```bash
|
4576
4643
|
ascli faspex5 admin res shared list --value=@json:'{"all":true}' --fields=id,name
|
4577
4644
|
```
|
4578
4645
|
|
4579
4646
|
- Create Metadata profile
|
4580
4647
|
|
4581
|
-
```
|
4648
|
+
```bash
|
4582
4649
|
ascli faspex5 admin res metadata_profiles create --value=@json:'{"name":"the profile","default":false,"title":{"max_length":200,"illegal_chars":[]},"note":{"max_length":400,"illegal_chars":[],"enabled":false},"fields":[{"ordering":0,"name":"field1","type":"text_area","require":true,"illegal_chars":[],"max_length":100},{"ordering":1,"name":"fff2","type":"option_list","require":false,"choices":["opt1","opt2"]}]}'
|
4583
4650
|
```
|
4584
4651
|
|
4585
4652
|
- Create a Shared inbox with specific metadata profile
|
4586
4653
|
|
4587
|
-
```
|
4654
|
+
```bash
|
4588
4655
|
ascli faspex5 admin res shared create --value=@json:'{"name":"the shared inbox","metadata_profile_id":1}'
|
4589
4656
|
```
|
4590
4657
|
|
4658
|
+
- List content in Shared folder and send package from remote source
|
4659
|
+
|
4660
|
+
```bash
|
4661
|
+
ascli faspex5 shared_folders list
|
4662
|
+
```
|
4663
|
+
|
4664
|
+
```bash
|
4665
|
+
ascli faspex5 shared_folders br 3 /folder
|
4666
|
+
```
|
4667
|
+
|
4668
|
+
```bash
|
4669
|
+
ascli faspex5 package send --value=@json:'{"title":"hello","recipients":[{"name":"_recipient_here_"}]}' --shared-folder=3 /folder/file
|
4670
|
+
```
|
4671
|
+
|
4672
|
+
### Faspex 4-style postprocessing script with Faspex 5
|
4673
|
+
|
4674
|
+
`ascli` provides command `postprocessing` in plugin `faspex5` to emulate Faspex 4 postprocessing.
|
4675
|
+
It implements Faspex 5 web hooks, and calls a local script with the same environment as Faspex 4.
|
4676
|
+
|
4677
|
+
It is invoked like this:
|
4678
|
+
|
4679
|
+
```bash
|
4680
|
+
ascli faspex5 postprocessing --value=@json:'{"url":"http://localhost:8080/processing"}'
|
4681
|
+
```
|
4682
|
+
|
4683
|
+
The following parameters are supported:
|
4684
|
+
|
4685
|
+
| parameter | type | default | description |
|
4686
|
+
|----------------------------|---------|------------------------|-----------------------------------------------------|
|
4687
|
+
| url | string | http://localhost:8080 | Defines the base url on which requests are listened |
|
4688
|
+
| certificate | hash | nil | used to define certificate if https is used |
|
4689
|
+
| certificate.key | string | nil | path to private key file |
|
4690
|
+
| certificate.cert | string | nil | path to certificate |
|
4691
|
+
| certificate.chain | string | nil | path to intermediary certificates |
|
4692
|
+
| processing | hash | nil | behaviour of post processing |
|
4693
|
+
| processing.script_folder | string | . | prefix added to script path |
|
4694
|
+
| processing.fail_on_error | bool | false | if true and process exit with non zero, then fail |
|
4695
|
+
| processing.timeout_seconds | integer | 60 | processing script is killed if takes more time |
|
4696
|
+
|
4697
|
+
Parameter `url` defines:
|
4698
|
+
|
4699
|
+
- if http or https is used
|
4700
|
+
- the local port
|
4701
|
+
- the "domain", i.e. main path of url
|
4702
|
+
|
4703
|
+
When a request is received the following happens:
|
4704
|
+
|
4705
|
+
- the processor get the path of the url called
|
4706
|
+
- it removes the "domain
|
4707
|
+
- it pre-prends it with the value of `script_folder`
|
4708
|
+
- it executes the script
|
4709
|
+
- upon success, a success code is returned
|
4710
|
+
|
4711
|
+
In Faspex 5, configure like this:
|
4712
|
+
|
4713
|
+
`Webhook endpoint URI` : `http://localhost:8080/processing/script1.sh`
|
4714
|
+
|
4715
|
+
Then, the postprocessing script executed will be `script1.sh`.
|
4716
|
+
|
4717
|
+
Environment variables at set to the values provided by the web hook which are the same as Faspex 4 postprocessing.
|
4718
|
+
|
4591
4719
|
## <a id="faspex"></a>Plugin: `faspex`: IBM Aspera Faspex v4
|
4592
4720
|
|
4593
4721
|
Notes:
|
4594
4722
|
|
4595
|
-
- The command
|
4723
|
+
- The command `v4` requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
|
4596
4724
|
- For full details on Faspex API, refer to: [Reference on Developer Site](https://developer.ibm.com/apis/catalog/?search=faspex)
|
4597
4725
|
|
4598
4726
|
### Listing Packages
|
@@ -4628,7 +4756,7 @@ If no parameter `max` or `pmax` is provided, then all packages will be listed in
|
|
4628
4756
|
|
4629
4757
|
#### Example: list packages in dropbox
|
4630
4758
|
|
4631
|
-
```
|
4759
|
+
```bash
|
4632
4760
|
ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'
|
4633
4761
|
```
|
4634
4762
|
|
@@ -4650,8 +4778,8 @@ ascli faspex package recv --link=faspe://...
|
|
4650
4778
|
If the package is in a specific **dropbox**/**workgroup**, add option `recipient` for both the `list` and `recv` commands.
|
4651
4779
|
|
4652
4780
|
```bash
|
4653
|
-
ascli faspex package list --recipient='*
|
4654
|
-
ascli faspex package recv 125 --recipient='*
|
4781
|
+
ascli faspex package list --recipient='*dropbox_name'
|
4782
|
+
ascli faspex package recv 125 --recipient='*dropbox_name'
|
4655
4783
|
```
|
4656
4784
|
|
4657
4785
|
if `id` is set to `ALL`, then all packages are downloaded, and if option `once_only`is used, then a persistency file is created to keep track of already downloaded packages.
|
@@ -4663,7 +4791,7 @@ The contents of `delivery_info` is directly the contents of the `send` v3 [API o
|
|
4663
4791
|
|
4664
4792
|
Example:
|
4665
4793
|
|
4666
|
-
```
|
4794
|
+
```bash
|
4667
4795
|
ascli faspex package send --delivery-info=@json:'{"title":"my title","recipients":["laurent.martin.aspera@fr.ibm.com"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2
|
4668
4796
|
```
|
4669
4797
|
|
@@ -4681,7 +4809,7 @@ Like for any transfer, a notification can be sent by email using parameters: `no
|
|
4681
4809
|
|
4682
4810
|
Example:
|
4683
4811
|
|
4684
|
-
```
|
4812
|
+
```bash
|
4685
4813
|
ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["aspera.user1@gmail.com"]}' ~/Documents/Samples/200KB.1 --notif-to=aspera.user1@gmail.com --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'
|
4686
4814
|
```
|
4687
4815
|
|
@@ -4691,7 +4819,7 @@ In this example the notification template is directly provided on command line.
|
|
4691
4819
|
|
4692
4820
|
Example:
|
4693
4821
|
|
4694
|
-
```
|
4822
|
+
```bash
|
4695
4823
|
ascli faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
|
4696
4824
|
ascli faspex v4 dropbox list
|
4697
4825
|
ascli faspex v4 dropbox delete --id=36
|
@@ -4711,17 +4839,17 @@ my_faspex_conf:
|
|
4711
4839
|
username: admin
|
4712
4840
|
password: MyUserPassword
|
4713
4841
|
storage:
|
4714
|
-
|
4842
|
+
my_storage:
|
4715
4843
|
node: "@preset:my_faspex_node"
|
4716
|
-
path: /
|
4844
|
+
path: /mydir
|
4717
4845
|
my_faspex_node:
|
4718
4846
|
url: https://10.25.0.3:9092
|
4719
4847
|
username: node_faspex
|
4720
4848
|
password: MyNodePassword
|
4721
4849
|
```
|
4722
4850
|
|
4723
|
-
In this example, a faspex storage named
|
4724
|
-
under the docroot in
|
4851
|
+
In this example, a faspex storage named `my_storage` exists in Faspex, and is located
|
4852
|
+
under the docroot in `/mydir` (this must be the same as configured in Faspex).
|
4725
4853
|
The node configuration name is "my_faspex_node" here.
|
4726
4854
|
|
4727
4855
|
Note: the v4 API provides an API for nodes and shares.
|
@@ -4778,14 +4906,17 @@ Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and
|
|
4778
4906
|
### Shares 1 sample commands
|
4779
4907
|
|
4780
4908
|
```bash
|
4781
|
-
shares admin
|
4909
|
+
shares admin group list
|
4910
|
+
shares admin node list
|
4782
4911
|
shares admin share list --fields=-status,status_message
|
4783
|
-
shares admin share user_permissions 1
|
4784
|
-
shares admin user
|
4785
|
-
shares admin user
|
4912
|
+
shares admin share user_permissions 1 list
|
4913
|
+
shares admin user add --type=ldap --value=the_name
|
4914
|
+
shares admin user app_authorizations 1 modify --value=@json:'{"app_login":true}'
|
4915
|
+
shares admin user app_authorizations 1 show
|
4916
|
+
shares admin user import --type=saml --value=@json:'{"id":"the_id","name_id":"the_name"}'
|
4786
4917
|
shares admin user list
|
4787
|
-
shares admin user
|
4788
|
-
shares admin user share_permissions 1
|
4918
|
+
shares admin user share_permissions 1 list
|
4919
|
+
shares admin user share_permissions 1 show 1
|
4789
4920
|
shares health
|
4790
4921
|
shares repository browse /
|
4791
4922
|
shares repository delete my_shares_upload/testfile.bin
|
@@ -4873,29 +5004,33 @@ or using the IBM Cloud CLI:
|
|
4873
5004
|
|
4874
5005
|
```bash
|
4875
5006
|
ibmcloud resource service-keys
|
4876
|
-
ibmcloud resource service-key
|
5007
|
+
ibmcloud resource service-key _service_key_name_here_ --output JSON|jq '.[0].credentials'>$HOME/service_creds.json
|
4877
5008
|
```
|
4878
5009
|
|
4879
5010
|
(if you don't have `jq` installed, extract the structure as follows)
|
4880
5011
|
|
4881
5012
|
It consists in the following structure:
|
4882
5013
|
|
4883
|
-
|
5014
|
+
<!-- spellchecker: disable -->
|
5015
|
+
|
5016
|
+
```json
|
4884
5017
|
{
|
4885
|
-
"apikey": "
|
5018
|
+
"apikey": "my_api_key_here",
|
4886
5019
|
"cos_hmac_keys": {
|
4887
|
-
"access_key_id": "
|
4888
|
-
"secret_access_key": "
|
5020
|
+
"access_key_id": "my_access_key_here",
|
5021
|
+
"secret_access_key": "my_secret_here"
|
4889
5022
|
},
|
4890
5023
|
"endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints",
|
4891
|
-
"iam_apikey_description": "
|
4892
|
-
"iam_apikey_name": "
|
5024
|
+
"iam_apikey_description": "my_description_here",
|
5025
|
+
"iam_apikey_name": "my_key_name_here",
|
4893
5026
|
"iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
|
4894
5027
|
"iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxxx.....",
|
4895
5028
|
"resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/xxxxxxx....."
|
4896
5029
|
}
|
4897
5030
|
```
|
4898
5031
|
|
5032
|
+
<!-- spellchecker: enable -->
|
5033
|
+
|
4899
5034
|
The field `resource_instance_id` is for option `crn`
|
4900
5035
|
|
4901
5036
|
The field `apikey` is for option `apikey`
|
@@ -4950,7 +5085,7 @@ In this case, some of the `sync` parameters are fill from parameters of the rela
|
|
4950
5085
|
> **Note:** All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`).
|
4951
5086
|
>
|
4952
5087
|
> **Note:** Two JSON syntax are supported for option `sync_info`.
|
4953
|
-
> The first is same sync payload as specified on the `async` option `--conf` or in the latest node API, this is the
|
5088
|
+
> The first is same sync payload as specified on the `async` option `--conf` or in the latest node API, this is the preferred syntax and allows a single session definition.
|
4954
5089
|
> The second (legacy) is specific to `ascli` and allows definition of multiple sync sessions in a single command, although usually only one sync session is defined.
|
4955
5090
|
|
4956
5091
|
Documentation on Async node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22).
|
@@ -5059,6 +5194,7 @@ dnf install unoconv
|
|
5059
5194
|
|
5060
5195
|
- Amazon Linux
|
5061
5196
|
|
5197
|
+
<!-- spellchecker: disable -->
|
5062
5198
|
```bash
|
5063
5199
|
amazon-linux-extras enable libreoffice
|
5064
5200
|
yum clean metadata
|
@@ -5067,6 +5203,7 @@ wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
|
|
5067
5203
|
mv unoconv /usr/bin
|
5068
5204
|
chmod a+x /usr/bin/unoconv
|
5069
5205
|
```
|
5206
|
+
<!-- spellchecker: enable -->
|
5070
5207
|
|
5071
5208
|
### Configuration
|
5072
5209
|
|
@@ -5079,9 +5216,9 @@ The `xfer` user has a special protected shell: `aspshell`, so changing identity
|
|
5079
5216
|
```bash
|
5080
5217
|
su -s /bin/bash - xfer
|
5081
5218
|
|
5082
|
-
ascli config preset update
|
5219
|
+
ascli config preset update mypreviewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346
|
5083
5220
|
|
5084
|
-
ascli config preset set default preview
|
5221
|
+
ascli config preset set default preview mypreviewconf
|
5085
5222
|
```
|
5086
5223
|
|
5087
5224
|
Here we assume that Office file generation is disabled, else remove this option.
|
@@ -5126,6 +5263,7 @@ Lets first setup a script that will be used in the scheduler and set up the envi
|
|
5126
5263
|
|
5127
5264
|
Example of startup script `cron_ascli`, which sets the Ruby environment and adds some timeout protection:
|
5128
5265
|
|
5266
|
+
<!-- spellchecker: disable -->
|
5129
5267
|
```bash
|
5130
5268
|
#!/bin/bash
|
5131
5269
|
# set a timeout protection, just in case
|
@@ -5134,6 +5272,7 @@ case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
|
|
5134
5272
|
rvm use 2.6 --quiet
|
5135
5273
|
exec timeout ${tmout} ascli "${@}"
|
5136
5274
|
```
|
5275
|
+
<!-- spellchecker: enable -->
|
5137
5276
|
|
5138
5277
|
Here the cronjob is created for user `xfer`.
|
5139
5278
|
|
@@ -5302,19 +5441,19 @@ The `smtp` option is a hash table (extended value) with the following fields:
|
|
5302
5441
|
```bash
|
5303
5442
|
ascli config preset set smtp_google server smtp.google.com
|
5304
5443
|
ascli config preset set smtp_google username john@gmail.com
|
5305
|
-
ascli config preset set smtp_google password
|
5444
|
+
ascli config preset set smtp_google password my_password_here
|
5306
5445
|
```
|
5307
5446
|
|
5308
5447
|
or
|
5309
5448
|
|
5310
|
-
```
|
5311
|
-
ascli config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"
|
5449
|
+
```bash
|
5450
|
+
ascli config preset init smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"my_password_here"}'
|
5312
5451
|
```
|
5313
5452
|
|
5314
5453
|
or
|
5315
5454
|
|
5316
5455
|
```bash
|
5317
|
-
ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=
|
5456
|
+
ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=my_password_here
|
5318
5457
|
```
|
5319
5458
|
|
5320
5459
|
Set this configuration as global default, for instance:
|
@@ -5382,7 +5521,7 @@ It aims at simplifying the startup of a FASP session from a programmatic stand p
|
|
5382
5521
|
- common to Aspera Connect API (browser javascript startTransfer)
|
5383
5522
|
- easy to generate by using any third party language specific JSON library
|
5384
5523
|
|
5385
|
-
Hopefully, IBM integrates this
|
5524
|
+
Hopefully, IBM integrates this directly in `ascp`, and this tool is made redundant.
|
5386
5525
|
|
5387
5526
|
This makes it easy to integrate with any language provided that one can spawn a sub process, write to its STDIN, read from STDOUT, generate and parse JSON.
|
5388
5527
|
|
@@ -5416,7 +5555,7 @@ There are special "extended" [*transfer-spec*](#transferspec) parameters support
|
|
5416
5555
|
Create a file `session.json` with:
|
5417
5556
|
|
5418
5557
|
```json
|
5419
|
-
{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"
|
5558
|
+
{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"my_password_here","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}
|
5420
5559
|
````
|
5421
5560
|
|
5422
5561
|
Then start the session:
|
@@ -5431,9 +5570,9 @@ asession < session.json
|
|
5431
5570
|
|
5432
5571
|
This is particularly useful for a persistent session ( with the [*transfer-spec*](#transferspec) parameter: `"keepalive":true` )
|
5433
5572
|
|
5434
|
-
```
|
5573
|
+
```json
|
5435
5574
|
asession
|
5436
|
-
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"
|
5575
|
+
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"my_password_here","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
|
5437
5576
|
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
|
5438
5577
|
{"type":"DONE"}
|
5439
5578
|
```
|
@@ -5498,16 +5637,16 @@ The general idea is to rely on :
|
|
5498
5637
|
|
5499
5638
|
Interesting `ascp` features are found in its arguments: (see `ascp` manual):
|
5500
5639
|
|
5501
|
-
- `ascp` already takes care of sending only
|
5640
|
+
- `ascp` already takes care of sending only **new** files: option `-k 1,2,3` (`resume_policy`)
|
5502
5641
|
- `ascp` has some options to remove or move files after transfer: `--remove-after-transfer`, `--move-after-transfer`, `--remove-empty-directories` (`remove_after_transfer`, `move_after_transfer`, `remove_empty_directories`)
|
5503
5642
|
- `ascp` has an option to send only files not modified since the last X seconds: `--exclude-newer-than`, `--exclude-older-than` (`exclude_newer_than`,`exclude_older_than`)
|
5504
5643
|
- `--src-base` (`src_base`) if top level folder name shall not be created on destination
|
5505
5644
|
|
5506
|
-
Note
|
5507
|
-
|
5508
|
-
|
5509
|
-
|
5510
|
-
|
5645
|
+
> **Note:** `ascli` takes transfer parameters exclusively as a [*transfer-spec*](#transferspec), with `--ts` parameter.
|
5646
|
+
>
|
5647
|
+
> **Note:** Most, but not all, native `ascp` arguments are available as standard [*transfer-spec*](#transferspec) parameters.
|
5648
|
+
>
|
5649
|
+
> **Note:** Only for the [`direct`](#agt_direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided option `transfer_info` parameter `ascp_args`.
|
5511
5650
|
|
5512
5651
|
#### server side and configuration
|
5513
5652
|
|
@@ -5530,7 +5669,7 @@ ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345
|
|
5530
5669
|
|
5531
5670
|
The local folder (here, relative path: `source_hot`) is sent (upload) to an aspera server.
|
5532
5671
|
Source files are deleted after transfer.
|
5533
|
-
Growing files will be sent only once they don't grow anymore (based on an 8-second
|
5672
|
+
Growing files will be sent only once they don't grow anymore (based on an 8-second cool-off period).
|
5534
5673
|
If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs.
|
5535
5674
|
|
5536
5675
|
### Example: unidirectional synchronization (upload) to server
|
@@ -5585,7 +5724,7 @@ OK - [transfer:ok]
|
|
5585
5724
|
```
|
5586
5725
|
|
5587
5726
|
```bash
|
5588
|
-
ascli server health
|
5727
|
+
ascli server health asctl status --cmd_prefix='sudo ' --format=nagios
|
5589
5728
|
```
|
5590
5729
|
|
5591
5730
|
```output
|
@@ -5631,11 +5770,14 @@ See [CHANGELOG.md](CHANGELOG.md)
|
|
5631
5770
|
|
5632
5771
|
## History
|
5633
5772
|
|
5634
|
-
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares).
|
5773
|
+
When I joined Aspera, there was only one CLI: `ascp`, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares).
|
5774
|
+
Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API.
|
5775
|
+
Since all products relate to file transfers using FASP (`ascp`), I thought it would be interesting to have a unified CLI for transfers using FASP.
|
5776
|
+
Also, because there was already the `ascp` tool, I thought of an extended tool : `eascp.pl` which was accepting all `ascp` options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).
|
5635
5777
|
|
5636
5778
|
There were a few pitfalls:
|
5637
5779
|
|
5638
|
-
- The tool was written in the aging `perl` language while most Aspera application products (but the Transfer Server) are written in `ruby`.
|
5780
|
+
- The tool was written in the aging `perl` language while most Aspera web application products (but the Transfer Server) are written in `ruby`.
|
5639
5781
|
- The tool was only for transfers, but not able to call other products APIs
|
5640
5782
|
|
5641
5783
|
So, it evolved into `ascli`:
|
@@ -5645,6 +5787,9 @@ So, it evolved into `ascli`:
|
|
5645
5787
|
- supports transfers with multiple [Transfer Agents](#agents), that's why transfer parameters moved from `ascp` command line to [*transfer-spec*](#transferspec) (more reliable , more standard)
|
5646
5788
|
- `ruby` is consistent with other Aspera products
|
5647
5789
|
|
5790
|
+
Over the time, a supported command line tool `aspera` was developed in C++, it was later on deprecated.
|
5791
|
+
It had the advantage of being relatively easy to installed, as a single executable (well, still using `ascp`), but it was too limited IMHO, and lacked a lot of the features of this CLI.
|
5792
|
+
|
5648
5793
|
## Common problems
|
5649
5794
|
|
5650
5795
|
### Error: "Remote host is not who we expected"
|
@@ -5653,7 +5798,7 @@ Cause: `ascp` >= 4.x checks fingerprint of highest server host key, including EC
|
|
5653
5798
|
|
5654
5799
|
Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):
|
5655
5800
|
|
5656
|
-
```
|
5801
|
+
```bash
|
5657
5802
|
--ts=@json:'{"sshfp":null}'
|
5658
5803
|
```
|
5659
5804
|
|
@@ -5665,7 +5810,7 @@ References: ES-1944 in release notes of 4.1 and to [HSTS admin manual section "C
|
|
5665
5810
|
|
5666
5811
|
Some Ruby gems dependencies require compilation of native parts (C).
|
5667
5812
|
This also requires Ruby header files.
|
5668
|
-
If Ruby was installed as a Linux Packages, then also install ruby
|
5813
|
+
If Ruby was installed as a Linux Packages, then also install ruby development package:
|
5669
5814
|
`ruby-dev` ir `ruby-devel`, depending on distribution.
|
5670
5815
|
|
5671
5816
|
### ED255519 key not supported
|