asperalm 0.10.11 → 0.10.12

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 07561a15d0851e8b88023368c26bcca014eff59e9f04d69f425616505a69d0ca
4
- data.tar.gz: 0e84b0236a2d4483143ba6c4cb4598182c4a80b1037736e17b87b3f528494dc1
3
+ metadata.gz: 5c08eb4a6de98262dbddde09b526a9728e7313b3363b95ecb3c5ce64ab9cc138
4
+ data.tar.gz: 9dbf5f3d474e9dd09dda29172a92fbe1d01b8441dbbd5c62d0919a9d96f651d2
5
5
  SHA512:
6
- metadata.gz: 40b4a6e291592a0146e7e026145b099c5917568773c14e44e02b4f6218c3e228dff539fa0665e048b7c6157f4a2a68647a25aeb7f96af3fdd7425bd014b1d604
7
- data.tar.gz: 8cae37f98954c094ad3cec32e66b61248e34797792b9926bef2131c0da965a0a659091a3fb9f5cc03996d13c0281f1f80e7521473650e983fbb3d437f93b95a3
6
+ metadata.gz: 7bce68ed92cf7dfa6d00d83720fff0f3f5f630a15da681c85cc8d69203367435bdbf95e136686e88590ff006ffc86eb016cb128a83a20f6fe327a890db62d89c
7
+ data.tar.gz: 60da7ec0e7cd521bb035231a9b25476eb96d4ed5b142420ed33c66d5984796779e26f3f8a4fce1e47ce84714b4d14fc219632b5fa1ed7abc1fcf228ede94c47b
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Asperalm - A Ruby library for Aspera transfers and "Amelia", the _Multi Layer IBM Aspera_ Command Line Tool
2
2
 
3
- Version : 0.10.11
3
+ Version : 0.10.12
4
4
 
5
5
 
6
6
  _Laurent/2016-2018_
@@ -39,7 +39,7 @@ Once the gem is installed, `mlia` shall be accessible:
39
39
 
40
40
  ```bash
41
41
  $ mlia --version
42
- 0.10.11
42
+ 0.10.12
43
43
  ```
44
44
 
45
45
  ## First use
@@ -378,13 +378,21 @@ By default, a table output will display one line per entry, and columns. Dependi
378
378
 
379
379
  Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).
380
380
 
381
- The value of options and arguments can optionally be retrieved using one of the following "readers":
381
+ The extended value syntax is:
382
+
383
+ ```
384
+ <0 or more decoders><0 or 1 reader><nothing or some text value>
385
+ ```
386
+
387
+ The difference between reader and decoder is order and ordinality. Both act like a function of value on right hand side. Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.
388
+
389
+ The following "readers" are supported:
382
390
 
383
391
  * @val:VALUE , prevent further special prefix processing, e.g. `--username=@val:laurent` sets the option `username` to value `laurent`.
384
392
  * @file:PATH , read value from a file (prefix "~/" is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
385
393
  * @path:PATH , performs path expansion (prefix "~/" is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
386
394
  * @env:ENVVAR , read from a named env var, e.g.--password=@env:MYPASSVAR
387
- * @stdin: , read from stdin
395
+ * @stdin: , read from stdin (no value on right)
388
396
  * @preset:NAME , get whole option preset value by name
389
397
 
390
398
  In addition it is possible to decode a value, using one or multiple decoders :
@@ -395,6 +403,7 @@ In addition it is possible to decode a value, using one or multiple decoders :
395
403
  * @ruby: execute ruby code
396
404
  * @csvt: decode a titled CSV value
397
405
  * @lines: split a string in multiple lines and return an array
406
+ * @incps: include values of presets specified by key include_presets in hash
398
407
 
399
408
  To display the result of an extended value, use the `config echo` command.
400
409
 
@@ -426,6 +435,15 @@ $ mlia config echo @csvt:@file:test.csv
426
435
  :......:.....................:
427
436
  ```
428
437
 
438
+ Example: create a hash and include values from preset named "config" of config file
439
+
440
+ ```
441
+ $ mlia config echo @incps:@json:'{"hello":true,"incps":["config"]}'
442
+ {"version"=>"0.9", "hello"=>true}
443
+ ```
444
+
445
+ Note that `@incps:@json:'{"incps":["config"]}'` or `@incps:@ruby:'{"incps"=>["config"]}'` is equivalent to: `@preset:config`
446
+
429
447
  ## <a name="native"></a>Structured Value
430
448
 
431
449
  Some options and parameters expect a _Structured Value_, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.
@@ -838,7 +856,7 @@ will effectively push files to the related server from the agent node.
838
856
 
839
857
  ### <a name="direct"></a>Direct (local ascp using FASPManager API)
840
858
 
841
- By default the CLI will use a local FASP protocol.
859
+ By default the CLI will use a local FASP protocol, equivalent to specifying `--transfer=direct`.
842
860
  `mlia` will detect locally installed Aspera products.
843
861
  Refer to section [FASP](#client).
844
862
 
@@ -848,6 +866,15 @@ To specify a FASP proxy (only supported with the `direct` agent), set the approp
848
866
  * `EX_http_proxy_url` (proxy for legacy http fallback)
849
867
  * `EX_ascp_args`
850
868
 
869
+ The `transfer-info` optionally provides the following auto resume parameters:
870
+
871
+ * iter_max (7)
872
+ * sleep_initial (2)
873
+ * sleep_factor (2)
874
+ * sleep_max (60)
875
+
876
+ This defines the resume algorithm.
877
+
851
878
  ### IBM Aspera Connect Client GUI
852
879
 
853
880
  By specifying option: `--transfer=connect`, `mlia` will start transfers
@@ -868,6 +895,10 @@ The `--transfer-info` parameter can directly specify a pre-configured [option pr
868
895
  `--transfer-info=@preset:<psetname>` or specified using the option syntax :
869
896
  `--transfer-info=@json:'{"url":"https://...","username":"theuser","password":"thepass"}'`
870
897
 
898
+ ### <a name="trinfoaoc"></a>Aspera on cloud
899
+
900
+ By specifying option: `--transfer=aoc`, WORK IN PROGRESS
901
+
871
902
  ### <a name="httpgw"></a>HTTP Gateway
872
903
 
873
904
  If it possible to send using a HTTP gateway, in case FASP is not allowed.
@@ -1150,11 +1181,11 @@ mlia -N server --url=my_url_here --username=my_username_here --ssh-keys=~/.ssh/i
1150
1181
  mlia -N server --url=my_url_here --username=my_username_here --ssh-keys=~/.ssh/id_rsa ctl all:status
1151
1182
  mlia -N server --url=my_url_here --username=my_username_here --ssh-keys=~/.ssh/id_rsa nodeadmin -- -l
1152
1183
  mlia -h
1184
+ mlia aspera admin ats access_key --id=akibmcloud --secret=my_secret_here node browse /
1153
1185
  mlia aspera admin ats access_key --id=akibmcloud delete
1154
- mlia aspera admin ats access_key --id=akibmcloud node browse /
1155
1186
  mlia aspera admin ats access_key create --cloud=aws --region=AWS_REGION --params=@json:'{"id":"ak_aws","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"'AWS_BUCKET'","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/"}}'
1156
- mlia aspera admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"id":"akibmcloud","name":"laurent key","storage":{"type":"ibm-s3","bucket":"ICOS_BUCKET","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/"}}'
1157
- mlia aspera admin ats access_key list --fields=name,id,secret
1187
+ mlia aspera admin ats access_key create --cloud=softlayer --region=ICOS_REGION --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"laurent key","storage":{"type":"ibm-s3","bucket":"ICOS_BUCKET","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/"}}'
1188
+ mlia aspera admin ats access_key list --fields=name,id
1158
1189
  mlia aspera admin ats cluster clouds
1159
1190
  mlia aspera admin ats cluster list
1160
1191
  mlia aspera admin ats cluster show --cloud=aws --region=AWS_REGION
@@ -1201,12 +1232,12 @@ mlia aspera user info modify @json:'{"name":"dummy change"}'
1201
1232
  mlia aspera user info show
1202
1233
  mlia aspera workspace
1203
1234
  mlia ats access_key --id=ak_aws delete
1204
- mlia ats access_key --id=akibmcloud cluster
1235
+ mlia ats access_key --id=akibmcloud --secret=my_secret_here cluster
1236
+ mlia ats access_key --id=akibmcloud --secret=my_secret_here node browse /
1205
1237
  mlia ats access_key --id=akibmcloud delete
1206
- mlia ats access_key --id=akibmcloud node browse /
1207
1238
  mlia ats access_key create --cloud=aws --region=AWS_REGION --params=@json:'{"id":"ak_aws","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"'AWS_BUCKET'","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/"}}'
1208
- mlia ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"id":"akibmcloud","name":"laurent key","storage":{"type":"ibm-s3","bucket":"ICOS_BUCKET","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/"}}'
1209
- mlia ats access_key list --fields=name,id,secret
1239
+ mlia ats access_key create --cloud=softlayer --region=ICOS_REGION --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"laurent key","storage":{"type":"ibm-s3","bucket":"ICOS_BUCKET","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/"}}'
1240
+ mlia ats access_key list --fields=name,id
1210
1241
  mlia ats api_key create
1211
1242
  mlia ats api_key instances
1212
1243
  mlia ats api_key list
@@ -1239,7 +1270,7 @@ mlia config plugins
1239
1270
  mlia config proxy_check --fpac=file:///./examples/proxy.pac https://eudemo.asperademo.com
1240
1271
  mlia console transfer current list
1241
1272
  mlia console transfer smart list
1242
- mlia console transfer smart sub 73 @json:'{"source":{"paths":["/AAA"]}}'
1273
+ mlia console transfer smart sub 112 @json:'{"source":{"paths":["10MB.1"]},"source_type":"user_selected"}'
1243
1274
  mlia faspex nagios_check
1244
1275
  mlia faspex package list
1245
1276
  mlia faspex package list --fields=package_id --format=csv --box=sent|tail -n 1)
@@ -1323,7 +1354,7 @@ mlia sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"re
1323
1354
  ```bash
1324
1355
  $ mlia -h
1325
1356
  NAME
1326
- mlia -- a command line tool for Aspera Applications (v0.10.11)
1357
+ mlia -- a command line tool for Aspera Applications (v0.10.12)
1327
1358
 
1328
1359
  SYNOPSIS
1329
1360
  mlia COMMANDS [OPTIONS] [ARGS]
@@ -1362,7 +1393,7 @@ OPTIONS: global
1362
1393
  -v, --version display version
1363
1394
  -w, --warnings check for language warnings
1364
1395
  --ui=ENUM method to start browser: text, graphical
1365
- --log-level=ENUM Log level: info, error, fatal, unknown, warn, debug
1396
+ --log-level=ENUM Log level: info, error, debug, fatal, unknown, warn
1366
1397
  --logger=ENUM log method: stderr, stdout, syslog
1367
1398
  --lock-port=VALUE prevent dual execution of a command, e.g. in cron
1368
1399
  --query=VALUE additional filter for API calls (extended value) (some commands)
@@ -1386,9 +1417,11 @@ OPTIONS:
1386
1417
  --fpac=VALUE proxy auto configuration URL
1387
1418
  -P, --presetVALUE load the named option preset from current config file
1388
1419
  --default=VALUE set as default configuration for specified plugin
1420
+ --secret=VALUE access key secret for node
1421
+ --secrets=VALUE access key secret for node
1389
1422
  --test-mode=ENUM skip user validation in wizard mode: yes, no
1390
1423
  --ts=VALUE override transfer spec values (Hash, use @json: prefix), current={}
1391
- --local-resume=VALUE set resume policy (Hash, use @json: prefix), current={:iter_max=>7, :sleep_initial=>2, :sleep_factor=>2, :sleep_max=>60}
1424
+ --local-resume=VALUE set resume policy (Hash, use @json: prefix), current=
1392
1425
  --to-folder=VALUE destination folder for downloaded files
1393
1426
  --sources=VALUE list of source files (see doc)
1394
1427
  --transfer-info=VALUE additional information for transfer client
@@ -1405,7 +1438,7 @@ OPTIONS:
1405
1438
 
1406
1439
 
1407
1440
  COMMAND: node
1408
- SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download nagios_check events space info license mkdir mklink mkfile rename delete search
1441
+ SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download api_details nagios_check events space info license mkdir mklink mkfile rename delete search
1409
1442
  OPTIONS:
1410
1443
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1411
1444
  --username=VALUE username to log in
@@ -1427,7 +1460,7 @@ OPTIONS:
1427
1460
 
1428
1461
 
1429
1462
  COMMAND: bss
1430
- SUBCOMMANDS: find show
1463
+ SUBCOMMANDS: subscription
1431
1464
  OPTIONS:
1432
1465
  --url=VALUE URL of application, e.g. https://org.asperafiles.com
1433
1466
  --username=VALUE username to log in
@@ -1458,12 +1491,9 @@ OPTIONS:
1458
1491
  --redirect-uri=VALUE API client redirect URI
1459
1492
  --private-key=VALUE RSA private key PEM value for JWT (prefix file path with @val:@file:)
1460
1493
  --workspace=VALUE name of workspace
1461
- --secret=VALUE access key secret for node
1462
- --secrets=VALUE access key secret for node
1463
1494
  --eid=VALUE identifier
1464
1495
  --name=VALUE resource name
1465
1496
  --link=VALUE public link to shared resource
1466
- --public-token=VALUE token value of public link
1467
1497
  --new-user-option=VALUE new user creation option
1468
1498
  --from-folder=VALUE share to share source folder
1469
1499
  --scope=VALUE scope for AoC API calls
@@ -1491,7 +1521,6 @@ OPTIONS:
1491
1521
  COMMAND: ats
1492
1522
  SUBCOMMANDS: cluster access_key api_key aws_trust_policy
1493
1523
  OPTIONS:
1494
- --secret=VALUE Access key secret
1495
1524
  --ibm-api-key=VALUE IBM API key, see https://cloud.ibm.com/iam/apikeys
1496
1525
  --instance=VALUE ATS instance in ibm cloud
1497
1526
  --ats-key=VALUE ATS key identifier (ats_xxx)
@@ -2088,6 +2117,48 @@ Then, transfer between those:
2088
2117
  $ mlia -Paoc_show aspera files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'
2089
2118
  ```
2090
2119
 
2120
+ * create registration key to register a node
2121
+ ```
2122
+ $ mlia aspera admin res admin/client create @json:'{"data":{"name":"laurenbt","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv
2123
+ jfqslfdjlfdjfhdjklqfhdkl
2124
+ ```
2125
+
2126
+ * delete all registration keys
2127
+
2128
+ ```
2129
+ $ mlia aspera admin res admin/client list --fields=id --format=csv|mlia aspera admin res admin/client delete --bulk=yes --id=@lines:@stdin:
2130
+ +-----+---------+
2131
+ | id | status |
2132
+ +-----+---------+
2133
+ | 99 | deleted |
2134
+ | 100 | deleted |
2135
+ | 101 | deleted |
2136
+ | 102 | deleted |
2137
+ +-----+---------+
2138
+ ```
2139
+
2140
+ ### Shared folders
2141
+
2142
+ BETA
2143
+
2144
+ * list shared folders in node
2145
+
2146
+ ```
2147
+ $ mlia aspera admin res node --id=8669 shared_folders
2148
+ ```
2149
+
2150
+ * list shared folders in workspace
2151
+
2152
+ ```
2153
+ $ mlia aspera admin res workspace --id=10818 shared_folders
2154
+ ```
2155
+
2156
+ * list members of shared folder
2157
+
2158
+ ```
2159
+ $ mlia aspera admin res node --id=8669 v4 perm 82 show
2160
+ ```
2161
+
2091
2162
  ### Send a Package
2092
2163
 
2093
2164
  Send a package:
@@ -2964,9 +3035,18 @@ So, it evolved into `mlia`:
2964
3035
 
2965
3036
  # Release Notes
2966
3037
 
3038
+ * 0.10.12
3039
+
3040
+ * added support for AoC node registration keys
3041
+ * replaced option : `local_resume` with `transfer_info` for agent `direct`
3042
+ * Transfer agent is no more a Singleton instance, but only one is used in CLI
3043
+ * `@incps` : new extended value modifier
3044
+ * ATS: no more provides access keys secrets: now user must provide it
3045
+ * begin work on "aoc" transfer agent
3046
+
2967
3047
  * 0.10.11
2968
3048
 
2969
- * minoir refactor and fixes
3049
+ * minor refactor and fixes
2970
3050
 
2971
3051
  * 0.10.10
2972
3052
 
@@ -49,7 +49,7 @@ assert_usage(!['-h','--help'].include?(ARGV.first),nil)
49
49
  # parse transfer spec
50
50
  begin
51
51
  transfer_spec_arg=ARGV.pop
52
- transfer_spec=Asperalm::Cli::ExtendedValue.instance.parse("transfer spec",transfer_spec_arg)
52
+ transfer_spec=Asperalm::Cli::ExtendedValue.instance.evaluate(transfer_spec_arg)
53
53
  rescue => e
54
54
  assert_usage(false,"Cannot extract transfer spec from: #{transfer_spec_arg}")
55
55
  end
@@ -66,7 +66,7 @@ if transfer_spec.has_key?(TS_TMP_FILELIST_FOLDER)
66
66
  transfer_spec.delete(TS_TMP_FILELIST_FOLDER)
67
67
  end
68
68
  # get local agent (ascp)
69
- client=Asperalm::Fasp::Local.instance
69
+ client=Asperalm::Fasp::Local.new
70
70
  # disable ascp output on stdout to not mix with JSON events
71
71
  client.quiet=true
72
72
  # display JSON instead of legacy Lines
@@ -23,14 +23,13 @@ Asperalm::Fasp::Installation.instance.config_folder = '.'
23
23
  #Asperalm::Fasp::Installation.instance.use_ascp_from_product('Aspera Connect')
24
24
 
25
25
  # get FASP Manager singleton based on above ascp location
26
- fasp_manager=Asperalm::Fasp::Local.instance
26
+ fasp_manager=Asperalm::Fasp::Local.new
27
27
 
28
28
  # Note that it would also be possible to start transfers using other agents
29
29
  #require 'asperalm/fasp/connect'
30
- #fasp_manager=Asperalm::Fasp::Connect.instance
31
- #fasp_manager=Asperalm::Fasp::Node.instance
30
+ #fasp_manager=Asperalm::Fasp::Connect.new
32
31
  #require 'asperalm/fasp/node'
33
- #Asperalm::Fasp::Node.instance.node_api=Asperalm::Rest.new()
32
+ #fasp_manager=Asperalm::Fasp::Node.new(Asperalm::Rest.new(...))
34
33
 
35
34
  ##############################################################
36
35
  # Optional : register an event listener
@@ -1 +1 @@
1
- 0.10.11
1
+ 0.10.12
@@ -30,50 +30,55 @@ module Asperalm
30
30
 
31
31
  def initialize
32
32
  @handlers={
33
- 'base64'=>{:type=>:decoder,:func=>lambda{|v|Base64.decode64(v)}},
34
- 'json' =>{:type=>:decoder,:func=>lambda{|v|JSON.parse(v)}},
35
- 'zlib' =>{:type=>:decoder,:func=>lambda{|v|Zlib::Inflate.inflate(v)}},
36
- 'ruby' =>{:type=>:decoder,:func=>lambda{|v|eval(v)}},
37
- 'csvt' =>{:type=>:decoder,:func=>lambda{|v|ExtendedValue.decode_csvt(v)}},
38
- 'lines' =>{:type=>:decoder,:func=>lambda{|v|v.split("\n")}},
39
- 'val' =>{:type=>:reader ,:func=>lambda{|v|v}},
40
- 'file' =>{:type=>:reader ,:func=>lambda{|v|File.read(File.expand_path(v))}},
41
- 'path' =>{:type=>:reader ,:func=>lambda{|v|File.expand_path(v)}},
42
- 'env' =>{:type=>:reader ,:func=>lambda{|v|ENV[v]}},
43
- 'stdin' =>{:type=>:reader ,:func=>lambda{|v|raise "no value allowed for stdin" unless v.empty?;STDIN.read}},
33
+ :decoder=>{
34
+ 'base64' =>lambda{|v|Base64.decode64(v)},
35
+ 'json' =>lambda{|v|JSON.parse(v)},
36
+ 'zlib' =>lambda{|v|Zlib::Inflate.inflate(v)},
37
+ 'ruby' =>lambda{|v|eval(v)},
38
+ 'csvt' =>lambda{|v|ExtendedValue.decode_csvt(v)},
39
+ 'lines' =>lambda{|v|v.split("\n")}
40
+ },
41
+ :reader=>{
42
+ 'val' =>lambda{|v|v},
43
+ 'file' =>lambda{|v|File.read(File.expand_path(v))},
44
+ 'path' =>lambda{|v|File.expand_path(v)},
45
+ 'env' =>lambda{|v|ENV[v]},
46
+ 'stdin' =>lambda{|v|raise "no value allowed for stdin" unless v.empty?;STDIN.read}
47
+ }
44
48
  # other handlers can be set using set_handler, e.g. preset is reader in config plugin
45
49
  }
46
50
  end
47
51
  public
48
52
 
49
- def modifiers;@handlers.keys;end
53
+ def modifiers;@handlers.keys.map{|i|@handlers[i].keys}.flatten;end
50
54
 
55
+ # add a new :reader or :decoder
56
+ # decoder can be chained, reader is last one on right
51
57
  def set_handler(name,type,method)
58
+ raise "type must be one of #{@handlers.keys}" unless @handlers.keys.include?(type)
52
59
  Log.log.debug("setting #{type} handler for #{name}")
53
- @handlers[name]={:type=>type,:func=>method}
60
+ @handlers[type][name]=method
54
61
  end
55
62
 
56
- # parse an option value if it is a String using supported extended vaklue modifiers
63
+ # parse an option value if it is a String using supported extended value modifiers
57
64
  # other value types are returned as is
58
- def parse(name_or_descr,value)
65
+ def evaluate(value)
59
66
  return value if !value.is_a?(String)
60
- decoder_list=@handlers.keys.select{|k|@handlers[k][:type].eql?(:decoder)}
61
- reader_list=@handlers.keys.select{|k|@handlers[k][:type].eql?(:reader)}
62
67
  # first determine decoders, in reversed order
63
68
  decoders_reversed=[]
64
- while (m=value.match(/^@([^:]+):(.*)/)) and decoder_list.include?(m[1])
69
+ while (m=value.match(/^@([^:]+):(.*)/)) and @handlers[:decoder].include?(m[1])
65
70
  decoders_reversed.unshift(m[1])
66
71
  value=m[2]
67
72
  end
68
73
  # then read value
69
- reader_list.each do |reader|
74
+ @handlers[:reader].each do |reader,method|
70
75
  if m=value.match(/^@#{reader}:(.*)/) then
71
- value=@handlers[reader][:func].call(m[1])
76
+ value=method.call(m[1])
72
77
  break
73
78
  end
74
79
  end
75
80
  decoders_reversed.each do |decoder|
76
- value=@handlers[decoder][:func].call(value)
81
+ value=@handlers[:decoder][decoder].call(value)
77
82
  end
78
83
  return value
79
84
  end # parse
@@ -152,11 +152,11 @@ module Asperalm
152
152
  print "#{type}: #{descr}> "
153
153
  entry=STDIN.gets.chomp
154
154
  break if entry.empty?
155
- result.push(ExtendedValue.instance.parse(descr,entry))
155
+ result.push(ExtendedValue.instance.evaluate(entry))
156
156
  end
157
157
  when :single
158
158
  print "#{type}: #{descr}> "
159
- result=ExtendedValue.instance.parse(descr,STDIN.gets.chomp)
159
+ result=ExtendedValue.instance.evaluate(STDIN.gets.chomp)
160
160
  else # one fixed
161
161
  print "#{expected.join(' ')}\n#{type}: #{descr}> "
162
162
  result=self.class.get_from_list(STDIN.gets.chomp,descr,expected)
@@ -178,9 +178,9 @@ module Asperalm
178
178
  # there are values
179
179
  case expected
180
180
  when :single
181
- result=ExtendedValue.instance.parse(descr,@unprocessed_cmd_line_arguments.shift)
181
+ result=ExtendedValue.instance.evaluate(@unprocessed_cmd_line_arguments.shift)
182
182
  when :multiple
183
- result = @unprocessed_cmd_line_arguments.shift(@unprocessed_cmd_line_arguments.length).map{|v|ExtendedValue.instance.parse(descr,v)}
183
+ result = @unprocessed_cmd_line_arguments.shift(@unprocessed_cmd_line_arguments.length).map{|v|ExtendedValue.instance.evaluate(v)}
184
184
  # if expecting list and only one arg of type array : it is the list
185
185
  if result.length.eql?(1) and result.first.is_a?(Array)
186
186
  result=result.first
@@ -223,7 +223,7 @@ module Asperalm
223
223
  raise "ERROR"
224
224
  #declare_option(option_symbol)
225
225
  end
226
- value=ExtendedValue.instance.parse(option_symbol,value)
226
+ value=ExtendedValue.instance.evaluate(value)
227
227
  Log.log.debug("set_option(#{where}) #{option_symbol}=#{value}")
228
228
  if @declared_options[option_symbol][:values].eql?(BOOLEAN_VALUES)
229
229
  value=enum_to_bool(value)
@@ -373,7 +373,7 @@ module Asperalm
373
373
  name=$1
374
374
  value=$2
375
375
  name.gsub!(OPTION_SEP_LINE,OPTION_SEP_NAME)
376
- value=ExtendedValue.instance.parse(name,value)
376
+ value=ExtendedValue.instance.evaluate(value)
377
377
  Log.log.debug("option #{name}=#{value}")
378
378
  result[name]=value
379
379
  @unprocessed_cmd_line_options.delete(optionval) if remove_from_remaining
@@ -13,10 +13,8 @@ module Asperalm
13
13
  module Plugins
14
14
  class Aspera < BasicAuthPlugin
15
15
  VAL_ALL='ALL'
16
- MAX_REDIRECT=10
17
- private_constant :VAL_ALL,:MAX_REDIRECT
16
+ private_constant :VAL_ALL
18
17
  attr_reader :api_aoc
19
- attr_accessor :option_ak_secret,:option_secrets
20
18
  def initialize(env)
21
19
  super(env)
22
20
  @default_workspace_id=nil
@@ -25,13 +23,9 @@ module Asperalm
25
23
  @persist_ids=nil
26
24
  @home_node_file=nil
27
25
  @api_aoc=nil
28
- @option_ak_secret=nil
29
- @option_secrets={}
30
26
  @url_token_data=nil
31
27
  @user_info=nil
32
- @ats=Ats.new(@agents.merge(skip_secret: true))
33
- self.options.set_obj_attr(:secret,self,:option_ak_secret)
34
- self.options.set_obj_attr(:secrets,self,:option_secrets)
28
+ @ats=Ats.new(@agents)
35
29
  self.options.add_opt_list(:auth,Oauth.auth_types,"type of Oauth authentication")
36
30
  self.options.add_opt_list(:operation,[:push,:pull],"client operation for transfers")
37
31
  self.options.add_opt_simple(:client_id,"API client identifier in application")
@@ -39,12 +33,9 @@ module Asperalm
39
33
  self.options.add_opt_simple(:redirect_uri,"API client redirect URI")
40
34
  self.options.add_opt_simple(:private_key,"RSA private key PEM value for JWT (prefix file path with @val:@file:)")
41
35
  self.options.add_opt_simple(:workspace,"name of workspace")
42
- self.options.add_opt_simple(:secret,"access key secret for node")
43
- self.options.add_opt_simple(:secrets,"access key secret for node")
44
36
  self.options.add_opt_simple(:eid,"identifier") # used ?
45
37
  self.options.add_opt_simple(:name,"resource name")
46
38
  self.options.add_opt_simple(:link,"public link to shared resource")
47
- self.options.add_opt_simple(:public_token,"token value of public link")
48
39
  self.options.add_opt_simple(:new_user_option,"new user creation option")
49
40
  self.options.add_opt_simple(:from_folder,"share to share source folder")
50
41
  self.options.add_opt_simple(:scope,"scope for AoC API calls")
@@ -52,15 +43,11 @@ module Asperalm
52
43
  self.options.set_option(:bulk,:no)
53
44
  self.options.set_option(:new_user_option,{'package_contact'=>true})
54
45
  self.options.set_option(:operation,:push)
55
- client_data=OnCloud.random_cli
56
46
  self.options.set_option(:auth,:jwt)
57
- self.options.set_option(:client_id,client_data.first)
58
- self.options.set_option(:client_secret,client_data.last)
59
47
  self.options.set_option(:scope,OnCloud::SCOPE_FILES_USER)
60
48
  self.options.set_option(:private_key,'@file:'+env[:private_key_path]) if env[:private_key_path].is_a?(String)
61
49
  self.options.parse_options!
62
50
  return if env[:man_only]
63
- raise CliBadArgument,"secrets shall be Hash" unless @option_secrets.is_a?(Hash)
64
51
  update_aoc_api
65
52
  end
66
53
 
@@ -69,7 +56,7 @@ module Asperalm
69
56
  def find_ak_secret(ak,mandatory=true)
70
57
  # secret hash is already provisioned
71
58
  # optionally override with specific secret
72
- @api_aoc.add_secrets({ak=>@option_ak_secret}) unless @option_ak_secret.nil?
59
+ @api_aoc.add_secrets({ak=>self.config.get_secret(ak,mandatory)})
73
60
  # check that secret was provided as single value or dictionary
74
61
  raise CliBadArgument,"Please provide option secret or entry in option secrets for: #{ak}" unless @api_aoc.has_secret(ak) or !mandatory
75
62
  end
@@ -169,21 +156,19 @@ module Asperalm
169
156
  end
170
157
  client_node_file = @api_aoc.resolve_node_file(client_home_node_file,client_folder)
171
158
  server_node_file = @api_aoc.resolve_node_file(server_home_node_file,server_folder)
172
- # force node as agent
173
- self.options.set_option(:transfer,:node)
174
- # force node api in node agent
175
- Fasp::Node.instance.node_api=@api_aoc.get_node_api(client_node_file[:node_info],OnCloud::SCOPE_NODE_USER)
159
+ # force node as transfer agent
160
+ @agents[:transfer].set_agent_instance(Fasp::Node.new(@api_aoc.get_node_api(client_node_file[:node_info],OnCloud::SCOPE_NODE_USER)))
176
161
  # additional node to node TS info
177
162
  add_ts={
178
163
  'remote_access_key' => server_node_file[:node_info]['access_key'],
179
164
  'destination_root_id' => server_node_file[:file_id],
180
165
  'source_root_id' => client_node_file[:file_id]
181
166
  }
182
- return Main.result_transfer(transfer_start(OnCloud::FILES,client_tr_oper,server_node_file,add_ts))
167
+ return Main.result_transfer(transfer_start(OnCloud::FILES_APP,client_tr_oper,server_node_file,add_ts))
183
168
  when :upload
184
169
  node_file = @api_aoc.resolve_node_file(top_node_file,self.transfer.destination_folder('send'))
185
170
  add_ts={'tags'=>{'aspera'=>{'files'=>{'parentCwd'=>"#{node_file[:node_info]['id']}:#{node_file[:file_id]}"}}}}
186
- return Main.result_transfer(transfer_start(OnCloud::FILES,'send',node_file,add_ts))
171
+ return Main.result_transfer(transfer_start(OnCloud::FILES_APP,'send',node_file,add_ts))
187
172
  when :download
188
173
  source_paths=self.transfer.ts_source_paths
189
174
  # special case for AoC : all files must be in same folder
@@ -198,7 +183,7 @@ module Asperalm
198
183
  # override paths with just filename
199
184
  add_ts={'tags'=>{'aspera'=>{'files'=>{'parentCwd'=>"#{node_file[:node_info]['id']}:#{node_file[:file_id]}"}}}}
200
185
  add_ts.merge!({'paths'=>source_paths})
201
- return Main.result_transfer(transfer_start(OnCloud::FILES,'receive',node_file,add_ts))
186
+ return Main.result_transfer(transfer_start(OnCloud::FILES_APP,'receive',node_file,add_ts))
202
187
  when :http_node_download
203
188
  source_paths=self.transfer.ts_source_paths
204
189
  source_folder=source_paths.shift['source']
@@ -229,101 +214,57 @@ module Asperalm
229
214
  fileid=self.options.get_next_argument('file id')
230
215
  node_file = @api_aoc.resolve_node_file(top_node_file)
231
216
  node_api=@api_aoc.get_node_api(node_file[:node_info],OnCloud::SCOPE_NODE_USER)
232
- items=node_api.read('permissions',{'include'=>['[]','access_level','permission_count'],'file_id'=>fileid,'inherited'=>false})[:data]
233
- return {:type=>:object_list,:data=>items}
217
+ command_perms=self.options.get_next_command([:show,:create])
218
+ case command_perms
219
+ when :show
220
+ items=node_api.read('permissions',{'include'=>['[]','access_level','permission_count'],'file_id'=>fileid,'inherited'=>false})[:data]
221
+ return {:type=>:object_list,:data=>items}
222
+ when :create
223
+ #value=self.options.get_next_argument('creation value')
224
+ set_workspace_info
225
+ access_id="ASPERA_ACCESS_KEY_ADMIN_WS_#{@workspace_id}"
226
+ node_file[:node_info]
227
+ params={
228
+ "file_id"=>fileid,
229
+ "access_type"=>"user",
230
+ "access_id"=>access_id,
231
+ "access_levels"=>["list","read","write","delete","mkdir","rename","preview"],
232
+ "tags"=>{
233
+ "aspera"=>{
234
+ "files"=>{
235
+ "workspace"=>{
236
+ "id"=>@workspace_id,
237
+ "workspace_name"=>@workspace_name,
238
+ "user_name"=>user_info['name'],
239
+ "shared_by_user_id"=>user_info['id'],
240
+ "shared_by_name"=>user_info['name'],
241
+ "shared_by_email"=>user_info['email'],
242
+ "shared_with_name"=>access_id,
243
+ "access_key"=>node_file[:node_info]['access_key'],
244
+ "node"=>node_file[:node_info]['name']}}}}}
245
+ item=node_api.create('permissions',params)[:data]
246
+ return {:type=>:single_object,:data=>item}
247
+ else raise "error"
248
+ end
234
249
  end # command_repo
235
250
  throw "ERR"
236
251
  end # execute_node_gen4_command
237
252
 
238
- # check option "link"
239
- # if present try to get token value (resolve redirection if short links used)
240
- # then set options url/token/auth
241
- def pub_link_to_url_auth_token
242
- public_link_url=self.options.get_option(:link,:optional)
243
- return if public_link_url.nil?
244
- # set to token if available after redirection
245
- url_param_token_pair=nil
246
- redirect_count=0
247
- loop do
248
- uri=URI.parse(public_link_url)
249
- if OnCloud::PATHS_PUBLIC_LINK.include?(uri.path)
250
- url_param_token_pair=URI::decode_www_form(uri.query).select{|e|e.first.eql?('token')}.first
251
- if url_param_token_pair.nil?
252
- raise CliBadArgument,"link option must be URL with 'token' parameter"
253
- end
254
- # ok we get it !
255
- self.options.set_option(:url,'https://'+uri.host)
256
- self.options.set_option(:public_token,url_param_token_pair.last)
257
- self.options.set_option(:auth,:url_token)
258
- return
259
- end
260
- Log.log.debug("no expected format: #{public_link_url}")
261
- raise "exceeded max redirection: #{MAX_REDIRECT}" if redirect_count > MAX_REDIRECT
262
- r = Net::HTTP.get_response(uri)
263
- if r.code.start_with?("3")
264
- public_link_url = r['location']
265
- raise "no location in redirection" if public_link_url.nil?
266
- Log.log.debug("redirect to: #{public_link_url}")
267
- else
268
- # not a redirection
269
- raise CliBadArgument,'not redirection, so link not supported'
270
- end
271
- end # loop
272
-
273
- raise CliBadArgument,'too many redirections'
274
- end
275
-
276
- def aoc_rest_params
277
-
278
- # if auth is a public link
279
- # option "link" is a shortcut for options: url, auth, public_token
280
- pub_link_to_url_auth_token
281
-
282
- # Connection paramaters (url and auth) to Aspera on Cloud
283
- # pre populate rest parameters based on URL
284
- res_rest_params=OnCloud.base_rest_params(self.options.get_option(:url,:mandatory))
285
- aoc_rest_auth=res_rest_params[:auth]
286
- aoc_rest_auth.merge!({
287
- :grant => self.options.get_option(:auth,:mandatory),
288
- :client_id => self.options.get_option(:client_id,:mandatory),
289
- :client_secret => self.options.get_option(:client_secret,:mandatory),
290
- :scope => self.options.get_option(:scope,:optional)
291
- })
292
-
293
- # add jwt payload for global ids
294
- if OnCloud.is_global_client_id?(aoc_rest_auth[:client_id])
295
- org=aoc_rest_auth[:base_url].gsub(/.*\//,'')
296
- aoc_rest_auth.merge!({:jwt_add=>{org: org}})
297
- end
298
-
299
- # fill other auth parameters based on Oauth method
300
- case aoc_rest_auth[:grant]
301
- when :web
302
- aoc_rest_auth.merge!({
303
- :redirect_uri => self.options.get_option(:redirect_uri,:mandatory)
304
- })
305
- when :jwt
306
- private_key_PEM_string=self.options.get_option(:private_key,:mandatory)
307
- aoc_rest_auth.merge!({
308
- :jwt_subject => self.options.get_option(:username,:mandatory),
309
- :jwt_private_key_obj => OpenSSL::PKey::RSA.new(private_key_PEM_string)
310
- })
311
- when :url_token
312
- aoc_rest_auth.merge!({
313
- :url_token => self.options.get_option(:public_token,:mandatory),
314
- })
315
- else raise "ERROR: unsupported auth method"
316
- end
317
- return res_rest_params
253
+ # build constructor option list for OnCloud based on options of CLI
254
+ def aoc_rest_params(subpath)
255
+ # copy command line options to args
256
+ opt=[:link,:url,:auth,:client_id,:client_secret,:scope,:redirect_uri,:private_key,:username].inject({}){|m,i|m[i]=self.options.get_option(i,:optional);m}
257
+ opt[:subpath]=subpath
258
+ return opt
318
259
  end
319
260
 
320
261
  # Create a new AoC API REST object and set @api_aoc.
321
262
  # Parameters based on command line options
322
263
  # @return nil
323
264
  def update_aoc_api
324
- @api_aoc=OnCloud.new(aoc_rest_params)
265
+ @api_aoc=OnCloud.new(aoc_rest_params('api/v1'))
325
266
  # add access key secrets
326
- @api_aoc.add_secrets(@option_secrets)
267
+ @api_aoc.add_secrets(self.config.get_secrets)
327
268
  return nil
328
269
  end
329
270
 
@@ -377,7 +318,7 @@ module Asperalm
377
318
  return nil
378
319
  end
379
320
 
380
- # @home_node_file (hash with :node_info and :file_id)
321
+ # @home_node_file (hash with :node_info and :file_id)
381
322
  def set_home_node_file
382
323
  if !@url_token_data.nil?
383
324
  assert_public_link_types(['view_shared_file'])
@@ -396,7 +337,7 @@ module Asperalm
396
337
  return nil
397
338
  end
398
339
 
399
- def do_bulk_operation(params,success,&do_action)
340
+ def do_bulk_operation(params,success,id_result='id',&do_action)
400
341
  params=[params] unless self.options.get_option(:bulk)
401
342
  raise "expecting Array" unless params.is_a?(Array)
402
343
  result=[]
@@ -408,7 +349,7 @@ module Asperalm
408
349
  one['status']=success
409
350
  result.push(one)
410
351
  end
411
- return {:type=>:object_list,:data=>result,:fields=>['id','status']}
352
+ return {:type=>:object_list,:data=>result,:fields=>[id_result,'status']}
412
353
  end
413
354
 
414
355
  # package creation params can give just email, and full hash is created
@@ -468,9 +409,7 @@ module Asperalm
468
409
  end
469
410
  when :subscription
470
411
  org=@api_aoc.read('organization')[:data]
471
- bss_rest_param=aoc_rest_params
472
- bss_rest_param[:base_url].gsub!('api/v1','bss/platform')
473
- bss_api=Rest.new(bss_rest_param)
412
+ bss_api=Rest.new(aoc_rest_params('bss/platform'))
474
413
  graphql_query="
475
414
  query ($organization_id: ID!) {
476
415
  aoc (organization_id: $organization_id) {
@@ -553,7 +492,7 @@ module Asperalm
553
492
  events=events_api.read("application_events")[:data]['application_events']
554
493
  return {:type=>:object_list,:data=>events}
555
494
  when :resource
556
- resource_type=self.options.get_next_argument('resource',[:self,:user,:group,:client,:contact,:dropbox,:node,:operation,:package,:saml_configuration, :workspace, :dropbox_membership,:short_link,:workspace_membership,'admin/apps_new'.to_sym])
495
+ resource_type=self.options.get_next_argument('resource',[:self,:user,:group,:client,:contact,:dropbox,:node,:operation,:package,:saml_configuration, :workspace, :dropbox_membership,:short_link,:workspace_membership,'admin/apps_new'.to_sym,'admin/client_registration_token'.to_sym])
557
496
  resource_class_path=resource_type.to_s+case resource_type;when :dropbox;'es';when :self,'admin/apps_new'.to_sym;'';else; 's';end
558
497
  singleton_object=[:self].include?(resource_type)
559
498
  global_operations=[:create,:list]
@@ -561,7 +500,7 @@ module Asperalm
561
500
  supported_operations.push(:modify,:delete,*global_operations) unless singleton_object
562
501
  supported_operations.push(:v4,:v3) if resource_type.eql?(:node)
563
502
  supported_operations.push(:set_pub_key) if resource_type.eql?(:client)
564
- supported_operations.push(:shared_folders) if resource_type.eql?(:workspace)
503
+ supported_operations.push(:shared_folders) if [:node,:workspace].include?(resource_type)
565
504
  command=self.options.get_next_command(supported_operations)
566
505
 
567
506
  # require identifier for non global commands
@@ -586,8 +525,12 @@ module Asperalm
586
525
  resource_instance_path=resource_class_path if singleton_object
587
526
  case command
588
527
  when :create
528
+ id_result='id'
529
+ id_result='token' if resource_class_path.eql?('admin/client_registration_tokens')
530
+ # TODO: report inconsistency: creation url is !=, and does not return id.
531
+ resource_class_path='admin/client_registration/token' if resource_class_path.eql?('admin/client_registration_tokens')
589
532
  list_or_one=self.options.get_next_argument("creation data (Hash)")
590
- return do_bulk_operation(list_or_one,'created')do|params|
533
+ return do_bulk_operation(list_or_one,'created',id_result)do|params|
591
534
  raise "expecting Hash" unless params.is_a?(Hash)
592
535
  @api_aoc.create(resource_class_path,params)[:data]
593
536
  end
@@ -598,8 +541,8 @@ module Asperalm
598
541
  when :node; default_fields.push('host','access_key')
599
542
  when :operation; default_fields=nil
600
543
  when :contact; default_fields=["email","name","source_id","source_type"]
601
- when 'admin/apps_new'.to_sym; list_query={:organization_apps=>true}
602
- default_fields=['app_type','available']
544
+ when 'admin/apps_new'.to_sym; list_query={:organization_apps=>true};default_fields=['app_type','available']
545
+ when 'admin/client_registration_token'.to_sym; default_fields=['id','value','data.client_subject_scopes','created_at']
603
546
  end
604
547
  result=@api_aoc.read(resource_class_path,url_query(list_query))
605
548
  self.format.display_status("Items: #{result[:data].length}/#{result[:http]['X-Total-Count']}")
@@ -643,8 +586,18 @@ module Asperalm
643
586
  # result.delete('token')
644
587
  # return { :type=>:single_object, :data =>result}
645
588
  when :shared_folders
646
- res_data=@api_aoc.read("#{resource_class_path}/#{res_id}/permissions")[:data]
647
- return { :type=>:object_list, :data =>res_data , :fields=>['id','node_name','file_id']} #
589
+ read_params = case resource_type
590
+ when :workspace;{'access_id'=>"ASPERA_ACCESS_KEY_ADMIN_WS_#{res_id}",'access_type'=>'user'}
591
+ when :node;{'include'=>['[]','access_level','permission_count'],'created_by_id'=>"ASPERA_ACCESS_KEY_ADMIN"}
592
+ else raise "error"
593
+ end
594
+ res_data=@api_aoc.read("#{resource_class_path}/#{res_id}/permissions",read_params)[:data]
595
+ fields=case resource_type
596
+ when :node;['id','file_id','file.path','access_type']
597
+ when :workspace;['id','node_id','file_id','node_name','file.path','tags.aspera.files.workspace.share_as']
598
+ else raise "error"
599
+ end
600
+ return { :type=>:object_list, :data =>res_data , :fields=>fields}
648
601
  else raise :ERROR
649
602
  end
650
603
  when :usage_reports
@@ -705,8 +658,8 @@ module Asperalm
705
658
 
706
659
  package_creation['workspace_id']=@workspace_id
707
660
 
708
- # list of files to include in package
709
- package_creation['file_names']=self.transfer.ts_source_paths.map{|i|File.basename(i['source'])}
661
+ # list of files to include in package, optional
662
+ #package_creation['file_names']=self.transfer.ts_source_paths.map{|i|File.basename(i['source'])}
710
663
 
711
664
  # lookup users
712
665
  resolve_package_recipients(package_creation,'recipients')
@@ -723,7 +676,7 @@ module Asperalm
723
676
 
724
677
  # execute transfer
725
678
  node_file = {node_info: node_info, file_id: package_info['contents_file_id']}
726
- return Main.result_transfer(transfer_start(OnCloud::PACKAGES,'send',node_file,OnCloud.package_tags(package_info,'upload')))
679
+ return Main.result_transfer(transfer_start(OnCloud::PACKAGES_APP,'send',node_file,OnCloud.package_tags(package_info,'upload')))
727
680
  when :recv
728
681
  if !@url_token_data.nil?
729
682
  assert_public_link_types(['view_received_package'])
@@ -756,7 +709,7 @@ module Asperalm
756
709
  self.format.display_status("downloading package: #{package_info['name']}")
757
710
  add_ts={'paths'=>[{'source'=>'.'}]}
758
711
  node_file = {node_info: node_info, file_id: package_info['contents_file_id']}
759
- statuses=transfer_start(OnCloud::PACKAGES,'receive',node_file,OnCloud.package_tags(package_info,'download').merge(add_ts))
712
+ statuses=transfer_start(OnCloud::PACKAGES_APP,'receive',node_file,OnCloud.package_tags(package_info,'download').merge(add_ts))
760
713
  result_transfer.push({'package'=>package_id,'status'=>statuses.map{|i|i.to_s}.join(',')})
761
714
  # update skip list only if all transfer sessions completed
762
715
  if TransferAgent.session_status(statuses).eql?(:success)