asperalm 0.9.36 → 0.10

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2681f3a08393a76fb5745aaba0a73a71dea06dd71231e84bceb6e54afa3683f8
4
- data.tar.gz: b61310d8e10d2fcf5cc583b671656a8993197148f25a031dcfb194e767fef0a4
3
+ metadata.gz: 6346649ffd52f45da5aafd73f160de12c07b1b46933684aad0236fd7db6e5a2b
4
+ data.tar.gz: ae0cf6683c8a2f06349bc31ec81212dfa44a5b7478c55c35a15b390af3cf549a
5
5
  SHA512:
6
- metadata.gz: 2ad9759c53723dbe3f085cca2e61db99e6f9c6f67ff83c60e4e377e720e5c194155a0d10114215ee263d622e75b2a12f0e376482ee0ab31b933a7a1a3ee5ac3d
7
- data.tar.gz: 6a9e3b9f1681e7cd6981aa2cc489fa808a7de225513e9cff27df89905c64f3cfa57c58e31d29b4c133d6e1c5ac067308a2c93cf914bdc9bb5666f0598867f853
6
+ metadata.gz: 95834287a62b014270d913ad342de71db5623351e14ac64fc9418c69868b37e445717217c202d3eb4c85787ae04fee1280bf14da4e2761b92017b7b6227ec99c
7
+ data.tar.gz: 27740aa33b88c3a1b98e9534d1fc354c2a683754c4cdbab2a46489f9aa4d2aa1ad65004002500f9ebcdfa17b123088d7097c4b0f122c087e0dd6ba06f98b9d1b
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Asperalm - A Ruby library for Aspera transfers and "Amelia", the _Multi Layer IBM Aspera_ Command Line Tool
2
2
 
3
- Version : 0.9.36
3
+ Version : 0.10
4
4
 
5
5
 
6
6
  _Laurent/2016-2018_
@@ -39,7 +39,7 @@ Once the gem is installed, `mlia` shall be accessible:
39
39
 
40
40
  ```bash
41
41
  $ mlia --version
42
- 0.9.36
42
+ 0.10
43
43
  ```
44
44
 
45
45
  ## First use
@@ -73,7 +73,7 @@ In order to make further calls more convenient, it is advised to define a [optio
73
73
  $ mlia config id demoserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
74
74
  updated: demoserver
75
75
  $ mlia config id default set server demoserver
76
- updated: default->server to demoserver
76
+ updated: default→server to demoserver
77
77
  $ mlia server browse /aspera-test-dir-large
78
78
  :............:...........:......:..............:...........................:............................:
79
79
  : zmode : zuid : zgid : size : mtime : name :
@@ -950,13 +950,13 @@ Note that this is different from the "ascp" command line. The paradigm used by `
950
950
 
951
951
  For ease of use and flexibility, the list of files to transfer is specified by the option `sources`. The accepted values are:
952
952
 
953
- * the literal `@args` (default value), in that case the list of files is directly provided at the end of the command line (see example above)
953
+ * the literal `@args` (default value), in that case the list of files is directly provided at the end of the command line (see at the beginning of this section).
954
954
 
955
955
  * an [Extended Value](#extended) holding an *Array of String*. Examples:
956
956
 
957
957
  ```
958
958
  --sources=@json:'["file1","file2"]'
959
- --sources=@lines:@stdin
959
+ --sources=@lines:@stdin:
960
960
  --sources=@ruby:'File.read("myfilelist").split("\n")'
961
961
  ```
962
962
 
@@ -1111,7 +1111,7 @@ mlia aspera files find / --value='\.partial$$'
1111
1111
  mlia aspera files http_node_download --to-folder=LOCAL_FOLDER /200KB.1
1112
1112
  mlia aspera files mkdir /testfolder
1113
1113
  mlia aspera files rename /testfolder newname
1114
- mlia aspera files transfer --from-folder=/ --to-folder=xxx 200KB.1
1114
+ mlia aspera files transfer --workspace=eudemo --from-folder='/Demo Files/aspera-test-dir-tiny' --to-folder=xxx 200KB.1
1115
1115
  mlia aspera files upload --to-folder=/ CLIENT_DEMOFILE_PATH
1116
1116
  mlia aspera files v3 info
1117
1117
  mlia aspera organization
@@ -1243,7 +1243,7 @@ mlia sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"re
1243
1243
  ```bash
1244
1244
  $ mlia -h
1245
1245
  NAME
1246
- mlia -- a command line tool for Aspera Applications (v0.9.36)
1246
+ mlia -- a command line tool for Aspera Applications (v0.10)
1247
1247
 
1248
1248
  SYNOPSIS
1249
1249
  mlia COMMANDS [OPTIONS] [ARGS]
@@ -1405,7 +1405,7 @@ OPTIONS:
1405
1405
 
1406
1406
 
1407
1407
  COMMAND: cos
1408
- SUBCOMMANDS: asp
1408
+ SUBCOMMANDS: node
1409
1409
  OPTIONS:
1410
1410
  --service-credentials=VALUE IBM Cloud service credentials (Hash)
1411
1411
  --region=VALUE IBM Cloud Object storage region
@@ -1905,15 +1905,44 @@ Download of files is straightforward with a specific syntax for the `aspera file
1905
1905
 
1906
1906
  ### Find Files
1907
1907
 
1908
- Use the command `aspera files find [--value=regex]`.
1908
+ The command `aspera files find [--value=expression]` will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)
1909
1909
 
1910
- if no regex is provided, it will list all files recursively.
1910
+ The expression can be of 3 formats:
1911
1911
 
1912
- if a regex is provided (using ruby regex syntax, very similar to standard syntax), it find files recursively whose
1913
- filename matches the regex.
1912
+ * empty (default) : all files, equivalent to: `exec:true`
1913
+ * not starting with `exec:` : the expression is a regular expression, using ruby regex syntax. equivalent to: `exec:f['name'].match(/expression/)`
1914
1914
 
1915
1915
  For instance, to find files with a special extension, use `--value='\.myext$'`
1916
1916
 
1917
+ * starting with `exec:` : the ruby code after the prefix is executed for each entry found. the entry variable name is `f`. the file is displayed if the result is true;
1918
+
1919
+ Examples of expressions: (think to prefix with `exec:` and put in single quotes using bash)
1920
+
1921
+ * find files more recent than 100 days
1922
+
1923
+ ```
1924
+ f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100
1925
+ ```
1926
+
1927
+ * expression to find files older than 1 year on a given node and store in file list
1928
+
1929
+ ```
1930
+ $ mlia aspera admin res node --name='my node name' --secret='my secret' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt
1931
+ ```
1932
+
1933
+ * delete the files, one by one
1934
+
1935
+ ```
1936
+ $ cat my_file_list.txt|while read path;do echo mlia aspera admin res node --name='my node name' --secret='my secret' v4 delete "$path" ;done
1937
+ ```
1938
+
1939
+ * delete the files in bulk
1940
+
1941
+ ```
1942
+ cat my_file_list.txt | mlia aspera admin res node --name='my node name' --secret='my secret' v3 delete @lines:@stdin:
1943
+ ```
1944
+
1945
+
1917
1946
  ## IBM Aspera High Speed Transfer Server (transfer)
1918
1947
 
1919
1948
  This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
@@ -2180,6 +2209,51 @@ delete all my access keys:
2180
2209
  for k in $(mlia ats access_key list --field=id --format=csv);do mlia ats access_key id $k delete;done
2181
2210
  ```
2182
2211
 
2212
+ ## IBM Cloud Object Storage
2213
+
2214
+ *BETA: experimental*
2215
+
2216
+ The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
2217
+
2218
+ Required information are:
2219
+
2220
+ * service credentials
2221
+ * region
2222
+ * bucket
2223
+
2224
+ Secrevice credentials are directly created using the IBM cloud web ui. Navigate to: Navigation Menu &rarr; Resource List &rarr; Cloud Object Storage &rarr; Storage &rarr; Cloud Object Storage &rarr; Service Credentials &rarr; &lt;select or create credentials&gt; &rarr; view credentials &rarr; copy
2225
+
2226
+ or using the CLI:
2227
+
2228
+ ```
2229
+ $ ibmcloud resource service-keys
2230
+ $ ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>service_creds.json
2231
+ ```
2232
+
2233
+ It consists in the following structure:
2234
+
2235
+ ```json
2236
+ {
2237
+ "apikey": "xxxxxxx.....",
2238
+ "cos_hmac_keys": {
2239
+ "access_key_id": "xxxxxxx.....",
2240
+ "secret_access_key": "xxxxxxx....."
2241
+ },
2242
+ "endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints",
2243
+ "iam_apikey_description": "my description ...",
2244
+ "iam_apikey_name": "my key name",
2245
+ "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
2246
+ "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxxx.....",
2247
+ "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/xxxxxxx....."
2248
+ }
2249
+ ```
2250
+
2251
+ Example:
2252
+
2253
+ ```
2254
+ mlia cos node --service-credentials=@json:@file:local/service_creds.json --region=us-south --bucket=laurent upload myfile.txt
2255
+ ```
2256
+
2183
2257
  ## IBM Aspera Sync
2184
2258
 
2185
2259
  A basic plugin to start an "async" using `mlia`. The main advantage is the possibility
@@ -2648,6 +2722,11 @@ Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue. Not
2648
2722
 
2649
2723
  # Release Notes
2650
2724
 
2725
+ * version 0.10
2726
+
2727
+ * support for transfer using IBM Cloud Object Storage
2728
+ * improved `find` action using arbitrary expressions
2729
+
2651
2730
  * version 0.9.36
2652
2731
 
2653
2732
  * added option to specify file pair lists
@@ -2715,7 +2794,7 @@ Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue. Not
2715
2794
  * version 0.9.20
2716
2795
 
2717
2796
  * improved wizard (prepare for AoC global client id)
2718
- * preview generator: addedoption : --skip-format=<png,mp4>
2797
+ * preview generator: addedoption : --skip-format=&lt;png,mp4&gt;
2719
2798
  * removed outdated pictures from this doc
2720
2799
 
2721
2800
  * version 0.9.19
@@ -2826,14 +2905,14 @@ Gems, or remove your ed25519 key from your `.ssh` folder to solve the issue. Not
2826
2905
 
2827
2906
  Breaking change:
2828
2907
 
2829
- * ats server list provisioned -> ats cluster list
2830
- * ats server list clouds -> ats cluster clouds
2831
- * ats server list instance --cloud=x --region=y -> ats cluster show --cloud=x --region=y
2832
- * ats server id xxx -> ats cluster show --id=xxx
2833
- * ats subscriptions -> ats credential subscriptions
2834
- * ats api_key repository list -> ats credential cache list
2835
- * ats api_key list -> ats credential list
2836
- * ats access_key id xxx -> ats access_key --id=xxx
2908
+ * ats server list provisioned &rarr; ats cluster list
2909
+ * ats server list clouds &rarr; ats cluster clouds
2910
+ * ats server list instance --cloud=x --region=y &rarr; ats cluster show --cloud=x --region=y
2911
+ * ats server id xxx &rarr; ats cluster show --id=xxx
2912
+ * ats subscriptions &rarr; ats credential subscriptions
2913
+ * ats api_key repository list &rarr; ats credential cache list
2914
+ * ats api_key list &rarr; ats credential list
2915
+ * ats access_key id xxx &rarr; ats access_key --id=xxx
2837
2916
 
2838
2917
  * Version 0.6.18
2839
2918
 
@@ -1 +1 @@
1
- 0.9.36
1
+ 0.10
@@ -40,7 +40,7 @@ module Asperalm
40
40
  'file' =>{:type=>:reader ,:func=>lambda{|v|File.read(File.expand_path(v))}},
41
41
  'path' =>{:type=>:reader ,:func=>lambda{|v|File.expand_path(v)}},
42
42
  'env' =>{:type=>:reader ,:func=>lambda{|v|ENV[v]}},
43
- 'stdin' =>{:type=>:reader ,:func=>lambda{|v|raise "no value allowed for stdin" unless v.empty?;STDIN.gets}},
43
+ 'stdin' =>{:type=>:reader ,:func=>lambda{|v|raise "no value allowed for stdin" unless v.empty?;STDIN.read}},
44
44
  # other handlers can be set using set_handler, e.g. preset is reader in config plugin
45
45
  }
46
46
  end
@@ -255,7 +255,7 @@ module Asperalm
255
255
  end
256
256
 
257
257
  def result_all_fields(results,table_rows_hash_val)
258
- raise "empty" if table_rows_hash_val.empty?
258
+ raise "cannot get field names for empty result" if table_rows_hash_val.empty?
259
259
  if table_rows_hash_val.is_a?(Array)
260
260
  # get the list of all column names used in all lines, not just frst one, as all lines may have different columns
261
261
  final_table_columns=table_rows_hash_val.inject({}){|m,v|v.keys.each{|c|m[c]=true};m}.keys
@@ -181,6 +181,10 @@ module Asperalm
181
181
  result=ExtendedValue.instance.parse(descr,@unprocessed_cmd_line_arguments.shift)
182
182
  when :multiple
183
183
  result = @unprocessed_cmd_line_arguments.shift(@unprocessed_cmd_line_arguments.length).map{|v|ExtendedValue.instance.parse(descr,v)}
184
+ # if expecting list and only one arg of type array : it is the list
185
+ if result.length.eql?(1) and result.first.is_a?(Array)
186
+ result=result.first
187
+ end
184
188
  else
185
189
  result=self.class.get_from_list(@unprocessed_cmd_line_arguments.shift,descr,expected)
186
190
  end
@@ -6,6 +6,7 @@ require 'asperalm/on_cloud'
6
6
  require 'asperalm/persistency_file'
7
7
  require 'securerandom'
8
8
  require 'resolv'
9
+ require 'date'
9
10
 
10
11
  module Asperalm
11
12
  module Cli
@@ -26,6 +27,7 @@ module Asperalm
26
27
  @api_aoc=nil
27
28
  @option_ak_secret=nil
28
29
  @url_token_data=nil
30
+ @user_info=nil
29
31
  @ats=Ats.new(@agents.merge(skip_secret: true))
30
32
  self.options.set_obj_attr(:secret,self,:option_ak_secret)
31
33
  self.options.add_opt_list(:auth,Oauth.auth_types,"type of Oauth authentication")
@@ -58,9 +60,25 @@ module Asperalm
58
60
  update_aoc_api
59
61
  end
60
62
 
63
+ def user_info
64
+ return @user_info unless @user_info.nil?
65
+ # get our user's default information
66
+ # self?embed[]=default_workspace&embed[]=organization
67
+ begin
68
+ @user_info=@api_aoc.read('self')[:data]
69
+ rescue
70
+ @user_info={
71
+ 'name' => 'unknown',
72
+ 'email' => 'unknown',
73
+ }
74
+ end
75
+ end
76
+
61
77
  # starts transfer using transfer agent
62
78
  def transfer_start(app,direction,node_file,ts_add)
63
- return self.transfer.start(*@api_aoc.tr_spec(app,direction,node_file,@workspace_id,@workspace_name,ts_add))
79
+ ts_add.deep_merge!(OnCloud.analytics_ts(app,direction,@workspace_id,@workspace_name))
80
+ ts_add.deep_merge!(OnCloud.console_ts(app,user_info['name'],user_info['email']))
81
+ return self.transfer.start(*@api_aoc.tr_spec(app,direction,node_file,ts_add))
64
82
  end
65
83
 
66
84
  NODE4_COMMANDS=[ :browse, :find, :mkdir, :rename, :delete, :upload, :download, :transfer, :http_node_download, :v3, :file, :bearer_token_node ]
@@ -87,10 +105,14 @@ module Asperalm
87
105
  return {:type=>:object_list,:data=>items,:fields=>['name','type','recursive_size','size','modified_time','access_level']}
88
106
  when :find
89
107
  thepath=self.options.get_next_argument('path')
90
- regex=self.options.get_option(:value,:optional)||''
108
+ exec_prefix="exec:"
109
+ expression=self.options.get_option(:value,:optional)||"#{exec_prefix}true"
91
110
  node_file=@api_aoc.resolve_node_file(top_node_file,thepath)
92
- # currently: test filename only
93
- test_block=lambda{|current_file_info|current_file_info['name'].match(/#{regex}/)}
111
+ if expression.start_with?(exec_prefix)
112
+ test_block=eval "lambda{|f|#{expression[exec_prefix.length..-1]}}"
113
+ else
114
+ test_block=lambda{|f|f['name'].match(/#{expression}/)}
115
+ end
94
116
  return {:type=>:object_list,:data=>@api_aoc.find_files(node_file,test_block),:fields=>['path']}
95
117
  when :mkdir
96
118
  thepath=self.options.get_next_argument('path')
@@ -135,10 +157,11 @@ module Asperalm
135
157
  # additional node to node TS info
136
158
  add_ts={
137
159
  'remote_access_key' => node_file_server[:node_info]['access_key'],
138
- #'destination_root_id' => node_file_server[:file_id],
160
+ 'destination_root_id' => node_file_server[:file_id],
139
161
  'source_root_id' => node_file_client[:file_id]
140
162
  }
141
- return Main.result_transfer(transfer_start(OnCloud::FILES,client_tr_oper,node_file_server,add_ts))
163
+ #return Main.result_transfer(transfer_start(OnCloud::FILES,client_tr_oper,node_file_server,add_ts))
164
+ return Main.result_transfer(transfer_start(OnCloud::FILES,client_tr_oper,node_file_client,add_ts))
142
165
  when :upload
143
166
  node_file = @api_aoc.resolve_node_file(top_node_file,self.transfer.destination_folder('send'))
144
167
  add_ts={'tags'=>{'aspera'=>{'files'=>{'parentCwd'=>"#{node_file[:node_info]['id']}:#{node_file[:file_id]}"}}}}
@@ -288,10 +311,8 @@ module Asperalm
288
311
  @default_workspace_id=@url_token_data['data']['workspace_id']
289
312
  @persist_ids=[] # TODO : @url_token_data['id'] ?
290
313
  else
291
- # get our user's default information
292
- self_data=@api_aoc.read('self')[:data]
293
- @default_workspace_id=self_data['default_workspace_id']
294
- @persist_ids=[self_data['id']]
314
+ @default_workspace_id=user_info['default_workspace_id']
315
+ @persist_ids=[user_info['id']]
295
316
  end
296
317
 
297
318
  ws_name=self.options.get_option(:workspace,:optional)
@@ -427,12 +448,11 @@ module Asperalm
427
448
  # return {:type=>:object_list,:data=>@api_aoc.read("client_settings/")[:data]}
428
449
  when :info
429
450
  command=self.options.get_next_command([ :show,:modify ])
430
- my_user=@api_aoc.read('self')[:data] # self?embed[]=default_workspace&embed[]=organization
431
451
  case command
432
452
  when :show
433
- return { :type=>:single_object, :data =>my_user }
453
+ return { :type=>:single_object, :data =>user_info }
434
454
  when :modify
435
- @api_aoc.update("users/#{my_user['id']}",self.options.get_next_argument('modified parameters (hash)'))
455
+ @api_aoc.update("users/#{user_info['id']}",self.options.get_next_argument('modified parameters (hash)'))
436
456
  return Main.result_status('modified')
437
457
  end
438
458
  end
@@ -0,0 +1,91 @@
1
+ require 'asperalm/cli/plugins/node'
2
+ require 'asperalm/cli/plugin'
3
+ require 'xmlsimple'
4
+
5
+ module Asperalm
6
+ module Cli
7
+ module Plugins
8
+ class Cos < Plugin
9
+ # IBM Cloud authentication
10
+ IBM_CLOUD_OAUTH_URL='https://iam.cloud.ibm.com/identity'
11
+ private_constant :IBM_CLOUD_OAUTH_URL
12
+ def initialize(env)
13
+ super(env)
14
+ @service_creds=nil
15
+ self.options.add_opt_simple(:service_credentials,'IBM Cloud service credentials (Hash)')
16
+ self.options.add_opt_simple(:region,'IBM Cloud Object storage region')
17
+ self.options.add_opt_simple(:bucket,'IBM Cloud Object storage bucket')
18
+ end
19
+ ACTIONS=[:node]
20
+
21
+ def execute_action
22
+ command=self.options.get_next_command(ACTIONS)
23
+ case command
24
+ when :node
25
+ # get service credentials, Hash, e.g. @json:@file:...
26
+ service_credentials=self.options.get_option(:service_credentials,:mandatory)
27
+ # check necessary contents
28
+ raise CliBadArgument,'service_credentials must be a Hash' unless service_credentials.is_a?(Hash)
29
+ ['apikey','endpoints','resource_instance_id'].each do |field|
30
+ raise CliBadArgument,"service_credentials must have a field: #{field}" unless service_credentials.has_key?(field)
31
+ end
32
+ Asperalm::Log.dump('service_credentials',service_credentials)
33
+ # get options
34
+ bucket_region=self.options.get_option(:region,:mandatory)
35
+ bucket_name=self.options.get_option(:bucket,:mandatory)
36
+ # get API key from service credentials
37
+ serv_cred_storage_api_key=service_credentials['apikey']
38
+ # read endpoints from service provided in service credentials
39
+ endpoints=Asperalm::Rest.new({:base_url=>service_credentials['endpoints']}).read('')[:data]
40
+ Asperalm::Log.dump('endpoints',endpoints)
41
+ storage_endpoint=endpoints['service-endpoints']['regional'][bucket_region]['public'][bucket_region]
42
+ s3_api=Asperalm::Rest.new({
43
+ :base_url => "https://#{storage_endpoint}",
44
+ :not_auth_codes => ['401','403'],
45
+ :headers => {'ibm-service-instance-id' => service_credentials['resource_instance_id']},
46
+ :auth => {
47
+ :type => :oauth2,
48
+ :base_url => IBM_CLOUD_OAUTH_URL,
49
+ :grant => :ibm_apikey,
50
+ :api_key => serv_cred_storage_api_key
51
+ }})
52
+ # read FASP connection information for bucket
53
+ xml_result_text=s3_api.call({:operation=>'GET',:subpath=>bucket_name,:headers=>{'Accept'=>'application/xml'},:url_params=>{'faspConnectionInfo'=>nil}})[:http].body
54
+ ats_info=XmlSimple.xml_in(xml_result_text, {'ForceArray' => false})
55
+ Asperalm::Log.dump('ats_info',ats_info)
56
+ # get delegated token
57
+ delegated_oauth=Oauth.new({
58
+ :type => :oauth2,
59
+ :base_url => IBM_CLOUD_OAUTH_URL,
60
+ :grant => :delegated_refresh,
61
+ :api_key => serv_cred_storage_api_key,
62
+ :token_field=> 'delegated_refresh_token'
63
+ })
64
+ # to be placed in rest call header and in transfer tags
65
+ aspera_storage_credentials={
66
+ 'type' => 'token',
67
+ 'token' => {'delegated_refresh_token'=>delegated_oauth.get_authorization().gsub(/^Bearer /,'')}
68
+ }
69
+ # transfer spec addition
70
+ add_ts={'tags'=>{'aspera'=>{'node'=>{'storage_credentials'=>aspera_storage_credentials}}}}
71
+ # set a general addon to transfer spec
72
+ # here we choose to use the add_request_param
73
+ #self.transfer.option_transfer_spec_deep_merge(add_ts)
74
+ api_node=Rest.new({
75
+ :base_url => ats_info['ATSEndpoint'],
76
+ :headers => {'X-Aspera-Storage-Credentials'=>JSON.generate(aspera_storage_credentials)},
77
+ :auth => {
78
+ :type => :basic,
79
+ :username => ats_info['AccessKey']['Id'],
80
+ :password => ats_info['AccessKey']['Secret']}})
81
+ command=self.options.get_next_command([:upload,:download,:info,:access_key])
82
+ #command=self.options.get_next_command(Node::ACTIONS)
83
+ #command=self.options.get_next_command(Node::COMMON_ACTIONS)
84
+ node_plugin=Node.new(@agents.merge(skip_basic_auth_options: true, node_api: api_node, add_request_param: add_ts))
85
+ return node_plugin.execute_action(command)
86
+ end
87
+ end
88
+ end
89
+ end
90
+ end
91
+ end
@@ -180,9 +180,9 @@ module Asperalm
180
180
  transfer_spec.delete('paths')
181
181
  return Main.result_transfer(self.transfer.start(transfer_spec,{:src=>:node_gen3}))
182
182
  when :download
183
- send_result=@api_node.create('files/download_setup',{
184
- :transfer_requests => [ { :transfer_request => {
185
- :paths => self.transfer.ts_source_paths } } ] } )[:data]
183
+ transfer_request = {:paths => self.transfer.ts_source_paths }
184
+ transfer_request.deep_merge!(@add_request_param)
185
+ send_result=@api_node.create('files/download_setup',{:transfer_requests => [ { :transfer_request => transfer_request } ] } )[:data]
186
186
  # only one request, so only one answer
187
187
  transfer_spec=send_result['transfer_specs'].first['transfer_spec']
188
188
  return Main.result_transfer(self.transfer.start(transfer_spec,{:src=>:node_gen3}))
@@ -22,7 +22,7 @@ module Asperalm
22
22
 
23
23
  # @param param_hash
24
24
  def initialize(param_hash,params_definition)
25
- @param_hash=param_hash.clone # shallow copy is sufficient
25
+ @param_hash=param_hash # keep reference so that it can be modified by caller before calling `process_params`
26
26
  @params_definition=params_definition
27
27
  @result_env={}
28
28
  @result_args=[]
@@ -3,6 +3,7 @@ require 'asperalm/rest'
3
3
  require 'asperalm/open_application'
4
4
  require 'securerandom'
5
5
  require 'singleton'
6
+ require 'tty-spinner'
6
7
 
7
8
  module Asperalm
8
9
  module Fasp
@@ -63,6 +64,7 @@ module Asperalm
63
64
  def wait_for_transfers_completion
64
65
  connect_activity_args={'aspera_connect_settings'=>{'app_id'=>@connect_app_id}}
65
66
  started=false
67
+ spinner=nil
66
68
  loop do
67
69
  result=@connect_api.create('transfers/activity',connect_activity_args)[:data]
68
70
  if result['transfers']
@@ -74,7 +76,12 @@ module Asperalm
74
76
  notify_listeners("emulated",{'Type'=>'DONE'})
75
77
  break
76
78
  when 'initiating'
77
- puts 'starting'
79
+ if spinner.nil?
80
+ spinner = TTY::Spinner.new("[:spinner] :title", format: :classic)
81
+ spinner.start
82
+ end
83
+ spinner.update(title: trdata['status'])
84
+ spinner.spin
78
85
  when 'running'
79
86
  #puts "running: sessions:#{trdata["sessions"].length}, #{trdata["sessions"].map{|i| i['bytes_transferred']}.join(',')}"
80
87
  if !started and trdata["bytes_expected"] != 0
@@ -39,9 +39,9 @@ module Asperalm
39
39
  transfer_spec['tags']['aspera']['xfer_id']=SecureRandom.uuid
40
40
  Log.log.debug "xfer id=#{transfer_spec['xfer_id']}"
41
41
  # TODO: useful ? node only ?
42
- transfer_spec['tags']['aspera']['xfer_retry']=3600
42
+ transfer_spec['tags']['aspera']['xfer_retry']||=3600
43
43
  end
44
- Log.log.debug("ts=#{transfer_spec}")
44
+ Log.dump('ts',transfer_spec)
45
45
  # add bypass keys when authentication is token
46
46
  if transfer_spec['authentication'].eql?("token")
47
47
  Installation.instance.add_bypass_keys(transfer_spec)
@@ -246,7 +246,7 @@ module Asperalm
246
246
  when 'ERROR'
247
247
  Log.log.error("code: #{last_status_event['Code']}")
248
248
  if last_status_event['Description'] =~ /bearer token/i
249
- Log.log.error("need to regenrate token".red)
249
+ Log.log.error("need to regenerate token".red)
250
250
  if !session.nil? and session[:options].is_a?(Hash) and session[:options].has_key?(:regenerate_token)
251
251
  # regenerate token here, expired, or error on it
252
252
  env_args[:env]['ASPERA_SCP_TOKEN']=session[:options][:regenerate_token].call(true)
@@ -1,6 +1,7 @@
1
1
  require 'asperalm/fasp/manager'
2
2
  require 'asperalm/log'
3
3
  require 'singleton'
4
+ require 'tty-spinner'
4
5
 
5
6
  module Asperalm
6
7
  module Fasp
@@ -36,6 +37,13 @@ module Asperalm
36
37
 
37
38
  # generic method
38
39
  def start_transfer(transfer_spec,options=nil)
40
+ if transfer_spec['tags'].is_a?(Hash) and transfer_spec['tags']['aspera'].is_a?(Hash)
41
+ transfer_spec['tags']['aspera']['xfer_retry']||=150
42
+ end
43
+ # optimisation in case of sending to the same node
44
+ if transfer_spec['remote_host'].eql?(URI.parse(node_api_.params[:base_url]).host)
45
+ transfer_spec['remote_host']='localhost'
46
+ end
39
47
  resp=node_api_.create('ops/transfers',transfer_spec)[:data]
40
48
  @transfer_id=resp['id']
41
49
  Log.log.debug("tr_id=#{@transfer_id}")
@@ -45,6 +53,7 @@ module Asperalm
45
53
  # generic method
46
54
  def wait_for_transfers_completion
47
55
  started=false
56
+ spinner=nil
48
57
  # lets emulate management events to display progress bar
49
58
  loop do
50
59
  # status is empty sometimes with status 200...
@@ -54,7 +63,13 @@ module Asperalm
54
63
  notify_listeners('emulated',{'Type'=>'DONE'})
55
64
  break
56
65
  when 'waiting','partially_completed'
57
- puts trdata['status']
66
+ if spinner.nil?
67
+ spinner = TTY::Spinner.new("[:spinner] :title", format: :classic)
68
+ spinner.start
69
+ end
70
+ spinner.update(title: trdata['status'])
71
+ spinner.spin
72
+ #puts trdata
58
73
  when 'running'
59
74
  #puts "running: sessions:#{trdata["sessions"].length}, #{trdata["sessions"].map{|i| i['bytes_transferred']}.join(',')}"
60
75
  if !started and trdata['precalc'].is_a?(Hash) and
@@ -97,13 +97,20 @@ module Asperalm
97
97
  raise Fasp::Error.new('required: password or ssh key (value or path)')
98
98
  end
99
99
 
100
+ # special cases
101
+ @job_spec.delete('source_root') if @job_spec.has_key?('source_root') and @job_spec['source_root'].empty?
102
+
103
+ # process parameters as specified in table
100
104
  @builder.process_params
101
105
 
102
106
  # symbol must be index of Installation.paths
103
- env_args[:ascp_version]=@builder.process_param('use_ascp4',:get_value) ? :ascp4 : :ascp
104
-
105
- # destination will be base64 encoded, put before path arguments
106
- @builder.add_command_line_options(['--dest64'])
107
+ if @builder.process_param('use_ascp4',:get_value)
108
+ env_args[:ascp_version] = :ascp4
109
+ else
110
+ env_args[:ascp_version] = :ascp
111
+ # destination will be base64 encoded, put before path arguments
112
+ @builder.add_command_line_options(['--dest64'])
113
+ end
107
114
 
108
115
  # use file list if there is storage defined for it.
109
116
  src_dst_list=@builder.process_param('paths',:get_value,:accepted_types=>Array,:mandatory=>!@job_spec.has_key?('keepalive'))
@@ -130,8 +137,12 @@ module Asperalm
130
137
  end
131
138
  # optional args, at the end to override previous ones (to allow override)
132
139
  @builder.add_command_line_options(@builder.process_param('EX_ascp_args',:get_value,:accepted_types=>Array))
133
- # destination, use base64 encoding (as defined previously: --dest64) MUST be last one
134
- @builder.add_command_line_options([Base64.strict_encode64(@builder.process_param('destination_root',:get_value,:accepted_types=>String,:mandatory=>true))])
140
+ # process destination folder
141
+ destination_folder = @builder.process_param('destination_root',:get_value,:accepted_types=>String,:mandatory=>true)
142
+ # ascp4 does not support base64 encoding of destination
143
+ destination_folder = Base64.strict_encode64(destination_folder) unless env_args[:ascp_version].eql?(:ascp4)
144
+ # destination MUST be last command line argument to ascp
145
+ @builder.add_command_line_options([destination_folder])
135
146
 
136
147
  @builder.add_env_args(env_args[:env],env_args[:args])
137
148
 
@@ -1,14 +1,9 @@
1
1
  require 'asperalm/open_application'
2
-
3
- #require 'asperalm/rest'
4
2
  require 'base64'
5
3
  require 'date'
6
4
  require 'socket'
7
5
  require 'securerandom'
8
6
 
9
- # for future use
10
- UNUSED_STATE='ABC'
11
-
12
7
  module Asperalm
13
8
  # implement OAuth 2 for the REST client and generate a bearer token
14
9
  # call get_authorization() to get a token.
@@ -20,12 +15,11 @@ module Asperalm
20
15
  TOKEN_FILE_PREFIX='token'
21
16
  TOKEN_FILE_SEPARATOR='_'
22
17
  TOKEN_FILE_SUFFIX='.txt'
23
- NO_SCOPE='noscope'
24
18
  WINDOWS_PROTECTED_CHAR=%r{[/:"<>\\\*\?]}
25
19
  JWT_NOTBEFORE_OFFSET=300
26
20
  JWT_EXPIRY_OFFSET=3600
27
21
  @@token_cache_folder='.'
28
- private_constant :TOKEN_FILE_PREFIX,:TOKEN_FILE_SEPARATOR,:TOKEN_FILE_SUFFIX,:NO_SCOPE,:WINDOWS_PROTECTED_CHAR,:JWT_NOTBEFORE_OFFSET,:JWT_EXPIRY_OFFSET
22
+ private_constant :TOKEN_FILE_PREFIX,:TOKEN_FILE_SEPARATOR,:TOKEN_FILE_SUFFIX,:WINDOWS_PROTECTED_CHAR,:JWT_NOTBEFORE_OFFSET,:JWT_EXPIRY_OFFSET
29
23
  def self.persistency_folder; @@token_cache_folder;end
30
24
 
31
25
  def self.persistency_folder=(v); @@token_cache_folder=v;end
@@ -55,7 +49,7 @@ module Asperalm
55
49
  # :jwt_subject
56
50
  # :path_authorize (default: 'authorize')
57
51
  # :path_token (default: 'token')
58
- # :scope
52
+ # :scope (optional)
59
53
  # :grant (one of returned by self.auth_types)
60
54
  # :url_token
61
55
  # :user_name
@@ -80,7 +74,7 @@ module Asperalm
80
74
  }})
81
75
  end
82
76
  @token_auth_api=Rest.new(rest_params)
83
- # key = scope value, e.g. user:all, or node.*
77
+ # key = scope value, e.g. user:all, or node.*, or nil
84
78
  # value = ruby structure of data of returned value
85
79
  @token_cache={}
86
80
  if @params.has_key?(:redirect_uri)
@@ -116,8 +110,8 @@ module Asperalm
116
110
  # get location of cache for token, using some unique filename
117
111
  def token_filepath(api_scope)
118
112
  oauth_uri=URI.parse(@params[:base_url])
119
- parts=[oauth_uri.host.downcase.gsub(/[^a-z]+/,'_'),oauth_uri.path.downcase.gsub(/[^a-z]+/,'_'),@params[:grant],api_scope]
120
- parts.push(api_scope) if !api_scope.nil?
113
+ parts=[oauth_uri.host.downcase.gsub(/[^a-z]+/,'_'),oauth_uri.path.downcase.gsub(/[^a-z]+/,'_'),@params[:grant]]
114
+ parts.push(api_scope) unless api_scope.nil?
121
115
  parts.push(@params[:user_name]) if @params.has_key?(:user_name)
122
116
  parts.push(@params[:url_token]) if @params.has_key?(:url_token)
123
117
  parts.push(@params[:api_key]) if @params.has_key?(:api_key)
@@ -157,13 +151,16 @@ module Asperalm
157
151
 
158
152
  # @param options : :scope and :refresh
159
153
  def get_authorization(options={})
160
- api_scope=options[:scope] || @params[:scope] || NO_SCOPE
154
+ # api scope can be overriden to get auth for other scope
155
+ api_scope=options[:scope] || @params[:scope]
156
+ # as it is optional in many place: create struct
157
+ p_scope={}
158
+ p_scope[:scope] = api_scope unless api_scope.nil?
159
+ p_client_id_and_scope=p_scope.clone
160
+ p_client_id_and_scope[:client_id] = @params[:client_id] if @params.has_key?(:client_id)
161
161
  use_refresh_token=options[:refresh]
162
162
  # file name for cache of token
163
163
  token_state_file=token_filepath(api_scope)
164
- client_id_and_scope={}
165
- client_id_and_scope[:client_id] = @params[:client_id] if @params.has_key?(:client_id)
166
- client_id_and_scope[:scope] = api_scope unless api_scope.eql?(NO_SCOPE)
167
164
 
168
165
  # if first time, try to read from file
169
166
  if ! @token_cache.has_key?(api_scope) then
@@ -195,10 +192,9 @@ module Asperalm
195
192
  # try to refresh
196
193
  # note: admin token has no refresh, and lives by default 1800secs
197
194
  # Note: scope is mandatory in Files, and we can either provide basic auth, or client_Secret in data
198
- resp=create_token_simple(client_id_and_scope.merge({
195
+ resp=create_token_simple(p_client_id_and_scope.merge({
199
196
  :grant_type =>'refresh_token',
200
- :refresh_token=>refresh_token,
201
- :state =>UNUSED_STATE})) # TODO: remove, not useful
197
+ :refresh_token=>refresh_token}))
202
198
  if resp[:http].code.start_with?('2') then
203
199
  # save only if success ?
204
200
  save_and_set_token_cache(api_scope,resp[:http].body,token_state_file)
@@ -216,7 +212,7 @@ module Asperalm
216
212
  check_code=SecureRandom.uuid
217
213
  login_page_url=Rest.build_uri(
218
214
  "#{@params[:base_url]}/#{@params[:path_authorize]}",
219
- client_id_and_scope.merge({
215
+ p_client_id_and_scope.merge({
220
216
  :response_type => 'code',
221
217
  :redirect_uri => @params[:redirect_uri],
222
218
  :client_secret => @params[:client_secret],
@@ -227,11 +223,10 @@ module Asperalm
227
223
  code=goto_page_and_get_code(login_page_url,check_code)
228
224
 
229
225
  # exchange code for token
230
- resp=create_token_simple(client_id_and_scope.merge({
226
+ resp=create_token_simple(p_client_id_and_scope.merge({
231
227
  :grant_type => 'authorization_code',
232
228
  :code => code,
233
- :redirect_uri => @params[:redirect_uri],
234
- :state => UNUSED_STATE
229
+ :redirect_uri => @params[:redirect_uri]
235
230
  }))
236
231
  when :jwt
237
232
  # https://tools.ietf.org/html/rfc7519
@@ -260,20 +255,17 @@ module Asperalm
260
255
 
261
256
  Log.log.debug("assertion=[#{assertion}]")
262
257
 
263
- resp=create_token_simple({
258
+ resp=create_token_simple(p_scope.merge({
264
259
  :grant_type => 'urn:ietf:params:oauth:grant-type:jwt-bearer',
265
- :assertion => assertion,
266
- :scope => api_scope
267
- })
260
+ :assertion => assertion
261
+ }))
268
262
  when :url_token
269
263
  # exchange url_token for bearer token
270
264
  resp=create_token_advanced({
271
265
  :json_params => {:url_token=>@params[:url_token]},
272
- :url_params => {
273
- :grant_type => 'url_token',
274
- :scope => api_scope,
275
- :state => UNUSED_STATE
276
- }})
266
+ :url_params => p_scope.merge({
267
+ :grant_type => 'url_token'
268
+ })})
277
269
  when :ibm_apikey
278
270
  resp=create_token_simple({
279
271
  'grant_type' => 'urn:ibm:params:oauth:grant-type:apikey',
@@ -294,11 +286,11 @@ module Asperalm
294
286
  :type => :basic,
295
287
  :username => @params[:user_name],
296
288
  :password => @params[:user_pass]},
297
- :json_params => client_id_and_scope.merge({:grant_type => 'password'}), #:www_body_params also works
289
+ :json_params => p_client_id_and_scope.merge({:grant_type => 'password'}), #:www_body_params also works
298
290
  })
299
291
  when :body_userpass
300
292
  # legacy, not used
301
- resp=create_token_simple(client_id_and_scope.merge({
293
+ resp=create_token_simple(p_client_id_and_scope.merge({
302
294
  :grant_type => 'password',
303
295
  :username => @params[:user_name],
304
296
  :password => @params[:user_pass]
@@ -31,6 +31,7 @@ module Asperalm
31
31
  # found in aspera CLI
32
32
  RANDOM_CLIENT='==gYL9VcwFnRwBVLPBVR1JlS50mbtR2dfVmYMRHcTF3QuFHa1F0SmtEZvxEWDZzTGJTeM10ZzZWQNJUbYd2NYl0X6BHbWRzZCFnTPZHbKR2ZDhHbQBjWq1GNHNnUz1GcyZmO05WZpx2YtkGbj1CbhJ2bsdmLhJXZwNXY'
33
33
  private_constant :RANDOM_DRIVE,:RANDOM_CLIENT
34
+
34
35
  def self.random_drive
35
36
  Base64.strict_decode64(RANDOM_DRIVE.reverse).split(':')
36
37
  end
@@ -107,7 +108,7 @@ module Asperalm
107
108
  end
108
109
 
109
110
  # add details to show in analytics
110
- def analytics_ts(app,direction,ws_id,ws_name)
111
+ def self.analytics_ts(app,direction,ws_id,ws_name)
111
112
  # translate transfer to operation
112
113
  operation=case direction
113
114
  when 'send'; 'upload'
@@ -129,10 +130,20 @@ module Asperalm
129
130
  }
130
131
  end
131
132
 
133
+ # build ts addon for IBM Aspera Console
134
+ def self.console_ts(app,user_name,user_email)
135
+ elements=[app,user_name,user_email].map{|e|Base64.strict_encode64(e)}
136
+ elements.unshift('aspera.aoc')
137
+ Log.dump('elem1'.bg_red,elements[1])
138
+ return {
139
+ 'cookie'=>elements.join(':')
140
+ }
141
+ end
142
+
132
143
  # build "transfer info", 2 elements array with:
133
144
  # - transfer spec for aspera on cloud, based on node information and file id
134
145
  # - source and token regeneration method
135
- def tr_spec(app,direction,node_file,ws_id,ws_name,ts_add)
146
+ def tr_spec(app,direction,node_file,ts_add)
136
147
  # prepare the rest end point is used to generate the bearer token
137
148
  token_generation_method=lambda {|do_refresh|self.oauth_token(scope: self.class.node_scope(node_file[:node_info]['access_key'],SCOPE_NODE_USER), refresh: do_refresh)}
138
149
  # prepare transfer specification
@@ -144,10 +155,11 @@ module Asperalm
144
155
  'aspera' => {
145
156
  'app' => app,
146
157
  'files' => {
147
- 'node_id' => node_file[:node_info]['id'],
158
+ 'node_id' => node_file[:node_info]['id'],
148
159
  }, # files
149
160
  'node' => {
150
161
  'access_key' => node_file[:node_info]['access_key'],
162
+ #'file_id' => ts_add['source_root_id']
151
163
  'file_id' => node_file[:file_id]
152
164
  } # node
153
165
  } # aspera
@@ -156,7 +168,6 @@ module Asperalm
156
168
  transfer_spec.merge!(tr_spec_remote_info(node_file[:node_info]))
157
169
  # add caller provided transfer spec
158
170
  transfer_spec.deep_merge!(ts_add)
159
- transfer_spec.deep_merge!(analytics_ts(app,direction,ws_id,ws_name))
160
171
  # additional information for transfer agent
161
172
  source_and_token_generator={
162
173
  :src => :node_gen4,
@@ -14,6 +14,8 @@ module Asperalm
14
14
  # values for preview_format : output format
15
15
  PREVIEW_FORMATS=[:png,:mp4]
16
16
 
17
+ attr_reader :conversion_type
18
+
17
19
  # @param src source file path
18
20
  # @param dst destination file path
19
21
  # @param mime_type optional mime type as provided by node api
@@ -71,6 +71,7 @@ module Asperalm
71
71
  @http_session=nil
72
72
  # default is no auth
73
73
  @params[:auth]||={:type=>:none}
74
+ @params[:not_auth_codes]||=['401']
74
75
  # translate old auth parameters, remove prefix, place in auth
75
76
  [:auth,:basic,:oauth].each do |p_sym|
76
77
  p_str=p_sym.to_s+'_'
@@ -212,7 +213,7 @@ module Asperalm
212
213
  end
213
214
  rescue RestCallError => e
214
215
  # not authorized: oauth token expired
215
- if ['401','403'].include?(result[:http].code.to_s) and call_data[:auth][:type].eql?(:oauth2)
216
+ if @params[:not_auth_codes].include?(result[:http].code.to_s) and call_data[:auth][:type].eql?(:oauth2)
216
217
  begin
217
218
  # try to use refresh token
218
219
  req['Authorization']=oauth_token(refresh: true)
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: asperalm
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.9.36
4
+ version: '0.10'
5
5
  platform: ruby
6
6
  authors:
7
7
  - Laurent Martin
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-10-16 00:00:00.000000000 Z
11
+ date: 2019-10-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: xml-simple
@@ -150,6 +150,20 @@ dependencies:
150
150
  - - "~>"
151
151
  - !ruby/object:Gem::Version
152
152
  version: '1.8'
153
+ - !ruby/object:Gem::Dependency
154
+ name: tty-spinner
155
+ requirement: !ruby/object:Gem::Requirement
156
+ requirements:
157
+ - - ">"
158
+ - !ruby/object:Gem::Version
159
+ version: '0.9'
160
+ type: :runtime
161
+ prerelease: false
162
+ version_requirements: !ruby/object:Gem::Requirement
163
+ requirements:
164
+ - - ">"
165
+ - !ruby/object:Gem::Version
166
+ version: '0.9'
153
167
  - !ruby/object:Gem::Dependency
154
168
  name: bundler
155
169
  requirement: !ruby/object:Gem::Requirement
@@ -236,6 +250,7 @@ files:
236
250
  - lib/asperalm/cli/plugins/ats.rb
237
251
  - lib/asperalm/cli/plugins/config.rb
238
252
  - lib/asperalm/cli/plugins/console.rb
253
+ - lib/asperalm/cli/plugins/cos.rb
239
254
  - lib/asperalm/cli/plugins/faspex.rb
240
255
  - lib/asperalm/cli/plugins/node.rb
241
256
  - lib/asperalm/cli/plugins/orchestrator.rb
@@ -281,7 +296,7 @@ files:
281
296
  - lib/asperalm/sync.rb
282
297
  - lib/asperalm/temp_file_manager.rb
283
298
  - lib/asperalm/uri_reader.rb
284
- homepage: http://www.asperasoft.com
299
+ homepage: https://ibm.com/aspera
285
300
  licenses:
286
301
  - Apache-2.0
287
302
  metadata: