asperalm 0.11.4 → 0.11.5

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 582dfad06e02a5077f2ac5133db89bf7050db030994de189c4e50f9362cb6692
4
- data.tar.gz: 9664752a9082689c8aaab7c77a1f925ac3c82396897899e3f2ce8459ca2a0dd0
3
+ metadata.gz: 50f6ea18118d1ddaf904ccaaa31b3354cd75fe8832f8aba29ea448c4f1c90b46
4
+ data.tar.gz: 315f98f748ab810134d7f21c7dfdb1bfdd607eb35b75d3088422a55655f00971
5
5
  SHA512:
6
- metadata.gz: ae43afcb322a7b703756238c325240624d387fa737da03f4b37571ed93fdcc5c87cf7c9488e38e3206c092d5296b531502a2b072fa9b53ca2056a4be0a742996
7
- data.tar.gz: d2f3570f56967dba60ee5b53340c5fb889759051f0e54a1de8c0a68dfc7837a15fd8ac02786bf7c5f6ab6b90cc6a5a714112733dbb258ab71a7a8c70ac69bc5e
6
+ metadata.gz: 919edfbe6566a0467e11a84e2d401e39806cf7ba0f86967831737f8fe03e9aa035a5ea357b1c570bf41d77cfd14a1e8dcd1095dabb10680fbf1f078a313cffa7
7
+ data.tar.gz: 5bb3b4cbb702c73c190d7bd94e35af7101e846e5235957cf314d30a4b97bbe6639131b1b2d13b706cbe8dfa8523298d30e00b5762eedc30223fa08fafd3332f4
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Asperalm - A Ruby library for Aspera transfers and "Amelia", the _Multi Layer IBM Aspera_ Command Line Tool
2
2
 
3
- Version : 0.11.4
3
+ Version : 0.11.5
4
4
 
5
5
 
6
6
  _Laurent/2016-2020_
@@ -39,7 +39,7 @@ Once the gem is installed, `mlia` shall be accessible:
39
39
 
40
40
  ```
41
41
  $ mlia --version
42
- 0.11.4
42
+ 0.11.5
43
43
  ```
44
44
 
45
45
  ## First use
@@ -177,6 +177,25 @@ gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//')
177
177
  yum remove -y ruby ruby-libs
178
178
  ```
179
179
 
180
+ ### Other Unixes: Aix, etc...
181
+
182
+ If your unix do not provide a pre-built ruby, you can get using one of those
183
+ [methods](https://www.ruby-lang.org/en/documentation/installation/)
184
+
185
+
186
+ For instance to build from source, and install in `/opt/ruby` :
187
+
188
+ ```
189
+ # wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz
190
+ # gzip -d ruby-2.7.2.tar.gz
191
+ # tar xvf ruby-2.7.2.tar
192
+ # cd ruby-2.7.2
193
+ # ./configure --prefix=/opt/ruby
194
+ # make ruby.imp
195
+ # make
196
+ # make install
197
+ ```
198
+
180
199
  ## <a name="the_gem"></a>`asperalm` gem
181
200
 
182
201
  Once you have Ruby and rights to install gems: Install the gem and its dependencies:
@@ -730,9 +749,9 @@ terminal.
730
749
 
731
750
  To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honoured by Ruby when calling REST APIs.
732
751
 
733
- ## Proxy auto config (BETA)
752
+ ## Proxy auto config
734
753
 
735
- BETA: The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file, by its URL for local FASP agent. Supported schemes are : http:, https: and file:.
754
+ The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file, by its URL for local FASP agent. Supported schemes are : http:, https: and file:.
736
755
 
737
756
  The PAC file can be tested with command: `config proxy_check` , example:
738
757
 
@@ -741,7 +760,7 @@ $ mlia config proxy_check --fpac=file:///./proxy.pac http://www.example.com
741
760
  PROXY proxy.example.com:8080
742
761
  ```
743
762
 
744
- This is not yet implemented to specify proxy, so use env vars.
763
+ This is not yet implemented to specify http proxy, so use `http_proxy` env vars.
745
764
 
746
765
  ## <a name="client"></a>FASP configuration
747
766
 
@@ -1377,7 +1396,7 @@ mlia sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"re
1377
1396
  ```
1378
1397
  $ mlia -h
1379
1398
  NAME
1380
- mlia -- a command line tool for Aspera Applications (v0.11.4)
1399
+ mlia -- a command line tool for Aspera Applications (v0.11.5)
1381
1400
 
1382
1401
  SYNOPSIS
1383
1402
  mlia COMMANDS [OPTIONS] [ARGS]
@@ -1523,6 +1542,7 @@ OPTIONS:
1523
1542
  --scope=VALUE scope for AoC API calls
1524
1543
  --notify=VALUE notify users that file was received
1525
1544
  --bulk=ENUM bulk operation: yes, no
1545
+ --default-ports=ENUM use standard FASP ports or get from node api: yes, no
1526
1546
 
1527
1547
 
1528
1548
  COMMAND: alee
@@ -2173,8 +2193,6 @@ $ mlia aspera admin res admin/client list --fields=id --format=csv|mlia aspera a
2173
2193
 
2174
2194
  ### Shared folders
2175
2195
 
2176
- BETA
2177
-
2178
2196
  * list shared folders in node
2179
2197
 
2180
2198
  ```
@@ -2294,6 +2312,11 @@ $ mlia aspera admin analytics transfers --once-only=yes --lock-port=123455 \
2294
2312
 
2295
2313
  Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. here the period is [date of previous execution]..[now].
2296
2314
 
2315
+ ### Using specific transfer ports
2316
+
2317
+ By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that. The option `default_ports` ([yes]/no) allows `mlia` to retrieve the server ports from an API call (download_setup) which reads the information from `aspera.conf` on the server.
2318
+
2319
+
2297
2320
  ## IBM Aspera High Speed Transfer Server (transfer)
2298
2321
 
2299
2322
  This plugin works at FASP level (SSH/ascp/ascmd) and does not use the node API.
@@ -2600,8 +2623,6 @@ for k in $(mlia ats access_key list --field=id --format=csv);do mlia ats access_
2600
2623
 
2601
2624
  ## IBM Cloud Object Storage
2602
2625
 
2603
- *BETA: experimental*
2604
-
2605
2626
  The IBM Cloud Object Storage provides the possibility to execute transfers using FASP.
2606
2627
 
2607
2628
  Required information are:
@@ -3120,6 +3141,15 @@ So, it evolved into `mlia`:
3120
3141
 
3121
3142
  # Release Notes
3122
3143
 
3144
+ * 0.11.5
3145
+
3146
+ * added option `default_ports` for AoC (see manual)
3147
+ * allow bulk delete in `aspera files` with option `bulk=yes`
3148
+ * fix getting connect versions
3149
+ * added section for Aix
3150
+ * support all ciphers for `local`ascp (incl. gcm, etc..)
3151
+ * added transfer spec param `apply_local_docroot` for `local`
3152
+
3123
3153
  * 0.11.4
3124
3154
 
3125
3155
  * possibility to give shared inbox name when sending a package (else use id and type)
@@ -1 +1 @@
1
- 0.11.4
1
+ 0.11.5
@@ -41,13 +41,16 @@ module Asperalm
41
41
  self.options.add_opt_simple(:scope,"scope for AoC API calls")
42
42
  self.options.add_opt_simple(:notify,"notify users that file was received")
43
43
  self.options.add_opt_boolean(:bulk,"bulk operation")
44
+ self.options.add_opt_boolean(:default_ports,"use standard FASP ports or get from node api")
44
45
  self.options.set_option(:bulk,:no)
46
+ self.options.set_option(:default_ports,:yes)
45
47
  self.options.set_option(:new_user_option,{'package_contact'=>true})
46
48
  self.options.set_option(:operation,:push)
47
49
  self.options.set_option(:auth,:jwt)
48
50
  self.options.set_option(:scope,OnCloud::SCOPE_FILES_USER)
49
51
  self.options.set_option(:private_key,'@file:'+env[:private_key_path]) if env[:private_key_path].is_a?(String)
50
52
  self.options.parse_options!
53
+ OnCloud.set_use_default_ports(self.options.get_option(:default_ports))
51
54
  return if env[:man_only]
52
55
  update_aoc_api
53
56
  end
@@ -136,10 +139,13 @@ module Asperalm
136
139
  return Main.result_status("renamed #{thepath} to #{newname}")
137
140
  when :delete
138
141
  thepath=self.options.get_next_argument('path')
139
- node_file = @api_aoc.resolve_node_file(top_node_file,thepath)
140
- node_api=@api_aoc.get_node_api(node_file[:node_info],OnCloud::SCOPE_NODE_USER)
141
- result=node_api.delete("files/#{node_file[:file_id]}")[:data]
142
- return Main.result_status("deleted: #{thepath}")
142
+ return do_bulk_operation(thepath,'deleted','path') do |thepath|
143
+ raise "expecting String (path), got #{thepath.class.name} (#{thepath})" unless thepath.is_a?(String)
144
+ node_file = @api_aoc.resolve_node_file(top_node_file,thepath)
145
+ node_api=@api_aoc.get_node_api(node_file[:node_info],OnCloud::SCOPE_NODE_USER)
146
+ result=node_api.delete("files/#{node_file[:file_id]}")[:data]
147
+ {'path'=>thepath}
148
+ end
143
149
  when :transfer
144
150
  # client side is agent
145
151
  # server side is protocol server
@@ -339,19 +345,22 @@ module Asperalm
339
345
  return nil
340
346
  end
341
347
 
342
- def do_bulk_operation(params,success,id_result='id',&do_action)
343
- params=[params] unless self.options.get_option(:bulk)
344
- raise "expecting Array" unless params.is_a?(Array)
345
- result=[]
346
- params.each do |p|
347
- one={'id'=>p}
348
- # todo: manage exception and display status by default
349
- res=do_action.call(p)
350
- one=res if p.is_a?(Hash)
351
- one['status']=success
352
- result.push(one)
348
+ def do_bulk_operation(ids_or_one,success_msg,id_result='id',&do_action)
349
+ ids_or_one=[ids_or_one] unless self.options.get_option(:bulk)
350
+ raise "expecting Array" unless ids_or_one.is_a?(Array)
351
+ result_list=[]
352
+ ids_or_one.each do |id|
353
+ one={id_result=>id}
354
+ begin
355
+ res=do_action.call(id)
356
+ one=res if id.is_a?(Hash) # if block returns a has, let's use this
357
+ one['status']=success_msg
358
+ rescue => e
359
+ one['status']=e.to_s
360
+ end
361
+ result_list.push(one)
353
362
  end
354
- return {:type=>:object_list,:data=>result,:fields=>[id_result,'status']}
363
+ return {:type=>:object_list,:data=>result_list,:fields=>[id_result,'status']}
355
364
  end
356
365
 
357
366
  # package creation params can give just email, and full hash is created
@@ -123,7 +123,7 @@ module Asperalm
123
123
  connect_versions_javascript=javascript[:http].body.gsub(/\r?\n\s*/,'')
124
124
  Log.log.debug("javascript=[\n#{connect_versions_javascript}\n]")
125
125
  # get javascript object only
126
- found=connect_versions_javascript.match(/AW.connectVersions = (.*);/)
126
+ found=connect_versions_javascript.match(/^.*? = (.*);/)
127
127
  raise CliError,'Problen when getting connect versions from internet' if found.nil?
128
128
  alldata=JSON.parse(found[1])
129
129
  @connect_versions=alldata['entries']
@@ -42,7 +42,7 @@ module Asperalm
42
42
  'remove_after_transfer' => { :type => :opt_without_arg}, #TODO: doc
43
43
  'remove_empty_source_directory'=> { :type => :opt_without_arg}, #TODO: doc
44
44
  # value params
45
- 'cipher' => { :type => :opt_with_arg, :option_switch=>'-c',:accepted_types=>String,:translate_values=>{'aes128'=>'aes128','aes-128'=>'aes128','aes192'=>'aes192','aes-192'=>'aes192','aes256'=>'aes256','aes-256'=>'aes256','none'=>'none'}},
45
+ 'cipher' => { :type => :opt_with_arg, :option_switch=>'-c',:accepted_types=>String,:encode=>lambda{|cipher|cipher.tr('-','')}},
46
46
  'resume_policy' => { :type => :opt_with_arg, :option_switch=>'-k',:accepted_types=>String,:default=>'sparse_csum',:translate_values=>{'none'=>0,'attrs'=>1,'sparse_csum'=>2,'full_csum'=>3}},
47
47
  'direction' => { :type => :opt_with_arg, :option_switch=>'--mode',:accepted_types=>String,:translate_values=>{'receive'=>'recv','send'=>'send'}},
48
48
  'remote_user' => { :type => :opt_with_arg, :option_switch=>'--user',:accepted_types=>String},
@@ -72,6 +72,7 @@ module Asperalm
72
72
  'EX_multi_session_part' => { :type => :opt_with_arg, :option_switch=>'-C',:accepted_types=>String},
73
73
  'EX_no_read' => { :type => :opt_without_arg, :option_switch=>'--no-read'},
74
74
  'EX_no_write' => { :type => :opt_without_arg, :option_switch=>'--no-write'},
75
+ 'EX_apply_local_docroot' => { :type => :opt_without_arg, :option_switch=>'--apply-local-docroot'},
75
76
  # TODO: manage those parameters, some are for connect only ? node api ?
76
77
  'target_rate_cap_kbps' => { :type => :ignore, :accepted_types=>Integer},
77
78
  'target_rate_percentage' => { :type => :ignore, :accepted_types=>String}, # -wf -l<rate>p
@@ -6,6 +6,8 @@ require 'base64'
6
6
  module Asperalm
7
7
  class OnCloud < Rest
8
8
  private
9
+ @@use_standard_ports = true
10
+
9
11
  PRODUCT_NAME='Aspera on Cloud'
10
12
  # Production domain of AoC
11
13
  PROD_DOMAIN='ibmaspera.com'
@@ -22,8 +24,13 @@ module Asperalm
22
24
  PATHS_PUBLIC_LINK=['/packages/public/receive','/packages/public/send','/files/public']
23
25
  JWT_AUDIENCE='https://api.asperafiles.com/api/v1/oauth2/token'
24
26
  OAUTH_API_SUBPATH='api/v1/oauth2'
27
+ DEFAULT_TSPEC_INFO={
28
+ 'remote_user' => 'xfer',
29
+ 'ssh_port' => 33001,
30
+ 'fasp_port' => 33001
31
+ }
25
32
 
26
- private_constant :PRODUCT_NAME,:PROD_DOMAIN,:MAX_REDIRECT,:DEFAULT_CLIENT,:CLIENT_RANDOM,:PATHS_PUBLIC_LINK,:JWT_AUDIENCE,:OAUTH_API_SUBPATH
33
+ private_constant :PRODUCT_NAME,:PROD_DOMAIN,:MAX_REDIRECT,:DEFAULT_CLIENT,:CLIENT_RANDOM,:PATHS_PUBLIC_LINK,:JWT_AUDIENCE,:OAUTH_API_SUBPATH,:DEFAULT_TSPEC_INFO
27
34
 
28
35
  public
29
36
  # various API scopes supported
@@ -69,6 +76,10 @@ module Asperalm
69
76
  return 'node.'+access_key+':'+scope
70
77
  end
71
78
 
79
+ def self.set_use_default_ports(val)
80
+ @@use_standard_ports=val
81
+ end
82
+
72
83
  # check option "link"
73
84
  # if present try to get token value (resolve redirection if short links used)
74
85
  # then set options url/token/auth
@@ -191,17 +202,6 @@ module Asperalm
191
202
  }}}}
192
203
  end
193
204
 
194
- # get transfer connection parameters
195
- def self.tr_spec_remote_info(node_info)
196
- #TODO: add option to request those parameters by calling /upload_setup on node api
197
- return {
198
- 'remote_user' => 'xfer',
199
- 'remote_host' => node_info['host'],
200
- 'fasp_port' => 33001, # TODO: always the case ? or use upload_setup get get info ?
201
- 'ssh_port' => 33001, # TODO: always the case ?
202
- }
203
- end
204
-
205
205
  # add details to show in analytics
206
206
  def self.analytics_ts(app,direction,ws_id,ws_name)
207
207
  # translate transfer to operation
@@ -260,7 +260,15 @@ module Asperalm
260
260
  } # aspera
261
261
  } # tags
262
262
  }
263
- transfer_spec.merge!(self.class.tr_spec_remote_info(node_file[:node_info]))
263
+ # add remote host info
264
+ if @@use_standard_ports
265
+ transfer_spec.merge!(DEFAULT_TSPEC_INFO)
266
+ transfer_spec['remote_host']=node_file[:node_info]['host']
267
+ else
268
+ # retrieve values from API
269
+ std_t_spec=get_node_api(node_file[:node_info],SCOPE_NODE_USER).create('files/download_setup',{:transfer_requests => [ { :transfer_request => {:paths => [ {"source"=>'/'} ] } } ] } )[:data]['transfer_specs'].first['transfer_spec']
270
+ ['remote_host','remote_user','ssh_port','fasp_port'].each {|i| transfer_spec[i]=std_t_spec[i]}
271
+ end
264
272
  # add caller provided transfer spec
265
273
  transfer_spec.deep_merge!(ts_add)
266
274
  # additional information for transfer agent
@@ -272,6 +280,7 @@ module Asperalm
272
280
  end
273
281
 
274
282
  # returns a node API for access key
283
+ # @param scope e.g. SCOPE_NODE_USER
275
284
  # no scope: requires secret
276
285
  # if secret provided beforehand: use it
277
286
  def get_node_api(node_info,node_scope=nil)
@@ -103,7 +103,8 @@ module Asperalm
103
103
  d_t_s=myself.result[:data]['transfer_specs']
104
104
  if d_t_s.is_a?(Array)
105
105
  d_t_s.each do |res|
106
- r_err=res['transfer_spec']['error']
106
+ #r_err=res['transfer_spec']['error']
107
+ r_err=res['error']
107
108
  if r_err.is_a?(Hash)
108
109
  myself.add_error("T8:node: *_setup","#{r_err['code']}: #{r_err['reason']}: #{r_err['user_message']}")
109
110
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: asperalm
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.11.4
4
+ version: 0.11.5
5
5
  platform: ruby
6
6
  authors:
7
7
  - Laurent Martin
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-10-16 00:00:00.000000000 Z
11
+ date: 2020-11-16 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: xml-simple