asperalm 0.10.3 → 0.10.4

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8941c4f4f8bf577171407dde803b7098c812c12c7041ffad0bdc67b955acfcd4
4
- data.tar.gz: 27677e7f2bbc862e020e5f638256718a46c31500eacb2e92ba385a66f9b6de58
3
+ metadata.gz: 14072100dc3a6c73b360fedb36b688e418ba8af7aae42b2ee1191f57836f87a9
4
+ data.tar.gz: ba623efb19591461daed1ca53f4e42c43953a5fe3c7ccc89a81a3767af45f9c1
5
5
  SHA512:
6
- metadata.gz: 0e0664db8786d433de03d4098836211f7f4f50dfa442fff0d1315f78b2f5b8d7066b56fc96a604bc472d00d9c81c36bd8060a0505347df715d3bbeb8de83d0f3
7
- data.tar.gz: d9709516f8604272db60282a9f780b4c5bee0f60479fd9c1770f4db03029a05bf73c9c54ef2c030ee1f2dd44e2cee4f5768d154bb57e87bff277d91f509dbf4b
6
+ metadata.gz: 409d0c2d6d3dee8e1bfabf092344076647f51e825ca8750ba2e8a38678990003065847a27b76057176d3f9dc120d8539bdd1410fc9eed6fc7218ccadc37026b3
7
+ data.tar.gz: ac2957ec82193ab65414b336dcfe15e166b5302d420301f996b1465878b1cedd8f0e0ae8571069c61df5435c9b7556ee189d0386a0616eedee2430e0c67f9a8e
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Asperalm - A Ruby library for Aspera transfers and "Amelia", the _Multi Layer IBM Aspera_ Command Line Tool
2
2
 
3
- Version : 0.10.3
3
+ Version : 0.10.4
4
4
 
5
5
 
6
6
  _Laurent/2016-2018_
@@ -39,7 +39,7 @@ Once the gem is installed, `mlia` shall be accessible:
39
39
 
40
40
  ```bash
41
41
  $ mlia --version
42
- 0.10.3
42
+ 0.10.4
43
43
  ```
44
44
 
45
45
  ## First use
@@ -667,7 +667,7 @@ A (forward) proxy may be used for two reasons:
667
667
 
668
668
  To specify a HTTP proxy, set the HTTP_PROXY environment variable (or HTTPS_PROXY), those are honoured by Ruby when calling REST APIs.
669
669
 
670
- To specify a FASP proxy with "local" agent, set the appropriate transfer spec parameter or ascp parameter in transfer spec. (not possible with node agent or connect agent).
670
+ To specify a FASP proxy with "local" agent, set the appropriate [_transfer-spec_](#transferspec) parameter or ascp parameter in [_transfer-spec_](#transferspec). (not possible with node agent or connect agent).
671
671
 
672
672
  The `fpac` option allows specification of a Proxy Auto Configuration (PAC) file, by its URL for local FASP agent. Supported schemes are : http:, https: and file:.
673
673
 
@@ -953,7 +953,7 @@ $ mlia -Pdemoserver server upload --sources=@args ~/mysample.file secondfile
953
953
 
954
954
  More advanced options are provided to adapt to various cases. In fact, list of files to transfer are conveyed using the [_transfer-spec_](#transferspec) using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).
955
955
 
956
- Note that this is different from the "ascp" command line. The paradigm used by `mlia` is: all transfer parameters are kept in transfer spec so that execution of a transfer is independent of the transfer agent. It is envisioned that, one day, ascp will accept a transfer spec directly.
956
+ Note that this is different from the "ascp" command line. The paradigm used by `mlia` is: all transfer parameters are kept in [_transfer-spec_](#transferspec) so that execution of a transfer is independent of the transfer agent. It is envisioned that, one day, ascp will accept a [_transfer-spec_](#transferspec) directly.
957
957
 
958
958
  For ease of use and flexibility, the list of files to transfer is specified by the option `sources`. The accepted values are:
959
959
 
@@ -973,7 +973,7 @@ For ease of use and flexibility, the list of files to transfer is specified by t
973
973
  --sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'
974
974
  ```
975
975
 
976
- * Although not recommended, because it applies *only* to the `local` transfer agent (i.e. bare ascp), it is possible to specify bare ascp arguments using the pseudo transfer spec parameter `EX_ascp_args`. In that case, one must specify a dummy list in the transfer spec, which will be overriden by the bare ascp command line provided.
976
+ * Although not recommended, because it applies *only* to the `local` transfer agent (i.e. bare ascp), it is possible to specify bare ascp arguments using the pseudo [_transfer-spec_](#transferspec) parameter `EX_ascp_args`. In that case, one must specify a dummy list in the [_transfer-spec_](#transferspec), which will be overriden by the bare ascp command line provided.
977
977
 
978
978
  ```
979
979
  --sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'
@@ -1259,7 +1259,7 @@ mlia sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"re
1259
1259
  ```bash
1260
1260
  $ mlia -h
1261
1261
  NAME
1262
- mlia -- a command line tool for Aspera Applications (v0.10.3)
1262
+ mlia -- a command line tool for Aspera Applications (v0.10.4)
1263
1263
 
1264
1264
  SYNOPSIS
1265
1265
  mlia COMMANDS [OPTIONS] [ARGS]
@@ -1361,7 +1361,7 @@ OPTIONS:
1361
1361
  --params=VALUE parameters hash table, use @json:{"param":"value"}
1362
1362
  --result=VALUE specify result value as: 'work step:parameter'
1363
1363
  --synchronous=ENUM work step:parameter expected as result: yes, no
1364
- /Users/laurent/workspace/Rubytools/asperalm/lib/asperalm/cli/plugins/aspera.rb:587: warning: assigned but unused variable - data
1364
+ /Users/laurent/workspace/Rubytools/asperalm/lib/asperalm/cli/plugins/aspera.rb:592: warning: assigned but unused variable - data
1365
1365
 
1366
1366
 
1367
1367
  COMMAND: aspera
@@ -1389,6 +1389,7 @@ OPTIONS:
1389
1389
  --private-key=VALUE RSA private key PEM value for JWT (prefix file path with @val:@file:)
1390
1390
  --workspace=VALUE name of workspace
1391
1391
  --secret=VALUE access key secret for node
1392
+ --secrets=VALUE access key secret for node
1392
1393
  --eid=VALUE identifier
1393
1394
  --name=VALUE resource name
1394
1395
  --link=VALUE public link to shared resource
@@ -1781,6 +1782,31 @@ other query parameters can be used:
1781
1782
 
1782
1783
  Refer to the AoC API for full list of query parameters.
1783
1784
 
1785
+ #### Access Key secrets
1786
+
1787
+ In order to access some administrative actions on "nodes" (in fact, access keys), the associated
1788
+ secret is required.
1789
+
1790
+ When doing such action, it is possible to provide the secret using the `secret` option.
1791
+
1792
+ It is also possible to provoide a set of secrets used regularly. This can be done using the `secrets` option. The value provided shall be a Hash, where keys are access key ids, and values is the associated secret.
1793
+
1794
+ First choose a repository name, for example `my_secrets`, and populate it like this:
1795
+
1796
+ ```
1797
+ $ mlia conf id my_secrets set 'access_key1' 'secret1'
1798
+ $ mlia conf id my_secrets set 'access_key2' 'secret2'
1799
+ ```
1800
+
1801
+ If you have already set a `config` global preset to preset `cli_default` (refer to earlier in documentation), then the repository can be read by default like this (note the prefix `@val:` to avoid the evaluation of prefix `@preset:`):
1802
+
1803
+ ```
1804
+ $ mlia conf id cli_default set secrets @val:@preset:my_secrets
1805
+ ```
1806
+
1807
+ A secret repository can always be selected at runtime using `--secrets=@preset:xxxx`, or `--secrets=@json:'{"accesskey1":"secret1"}'`
1808
+
1809
+
1784
1810
  #### Examples
1785
1811
 
1786
1812
  * Bulk creation
@@ -2629,6 +2655,8 @@ It aims at simplifying the startup of a FASP session from a programmatic stand p
2629
2655
  * common to Aspera Connect API (browser javascript startTransfer)
2630
2656
  * easy to generate by using any third party language specific JSON library
2631
2657
 
2658
+ Hopefully, IBM integrates this diectly in `ascp`, and this tool is made redundant.
2659
+
2632
2660
  This makes it easy to integrate with any language provided that one can spawn a sub process, write to its STDIN, read from STDOUT, generate and parse JSON.
2633
2661
 
2634
2662
  The tool expect one single argument: a [_transfer-spec_](#transferspec).
@@ -2639,6 +2667,13 @@ Note that if JSON is the format, one has to specify `@json:` to tell the tool to
2639
2667
 
2640
2668
  During execution, it generates all low level events, one per line, in JSON format on stdout.
2641
2669
 
2670
+ Note that there are special "extended" [_transfer-spec_](#transferspec) parameters supported by `asession`:
2671
+
2672
+ * `EX_loglevel` to change log level of the tool
2673
+ * `EX_file_list_folder` to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is `[system tmp folder]/[username]_asession_filelists`
2674
+
2675
+ Note that in addition, many "EX_" [_transfer-spec_](#transferspec) parameters are supported for the "local" transfer agent (used by `asession`), refer to section [_transfer-spec_](#transferspec).
2676
+
2642
2677
  ## Comparison of interfaces
2643
2678
 
2644
2679
  <table>
@@ -2661,7 +2696,7 @@ echo "${MY_TSPEC}"|asession
2661
2696
 
2662
2697
  `asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: `LongParameter` -&gt; `long_parameter`
2663
2698
 
2664
- This is particularly useful for a persistent session ( with the transfer spec parameter: `"keepalive":true` )
2699
+ This is particularly useful for a persistent session ( with the [_transfer-spec_](#transferspec) parameter: `"keepalive":true` )
2665
2700
 
2666
2701
  ```
2667
2702
  $ asession
@@ -2738,7 +2773,7 @@ Note that:
2738
2773
 
2739
2774
  * `mlia` takes transfer parameters exclusively as a transfer_spec, with `--ts` parameter.
2740
2775
  * not all native ascp arguments are available as standard transfer_spec parameters
2741
- * native ascp arguments can be provided with the transfer spec parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
2776
+ * native ascp arguments can be provided with the [_transfer-spec_](#transferspec) parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)
2742
2777
 
2743
2778
  ### server side and configuration
2744
2779
 
@@ -2815,7 +2850,12 @@ As a workaround use another option, if available, to identify the entity, e.g. i
2815
2850
 
2816
2851
  # Release Notes
2817
2852
 
2818
- * version 0.10.2
2853
+ * version 0.10.4
2854
+
2855
+ * new options for AoC : `secrets`
2856
+ * ACLI-533 temp file list folder to use file lists is set by default, and used by asession
2857
+
2858
+ * version 0.10.3
2819
2859
 
2820
2860
  * included user name in oauth bearer token cache for AoC when JWT is used.
2821
2861
 
@@ -2953,7 +2993,7 @@ As a workaround use another option, if available, to identify the entity, e.g. i
2953
2993
 
2954
2994
  * version 0.9.7
2955
2995
 
2956
- * homogeneous transfer spec for node and local
2996
+ * homogeneous [_transfer-spec_](#transferspec) for node and local
2957
2997
  * preview persistency goes to unique file by default
2958
2998
  * catch mxf extension in preview as video
2959
2999
  * Faspex: possibility to download all paclages by specifying id=ALL
@@ -6,8 +6,10 @@ require 'asperalm/cli/listener/line_dump'
6
6
  require 'asperalm/cli/extended_value'
7
7
  require 'asperalm/log'
8
8
  require 'json'
9
-
9
+ # extended transfer spec parameter to change log level
10
10
  TS_LOGLEVEL='EX_loglevel'
11
+ # by default go to /tmp/username.filelist
12
+ TS_TMP_FILELIST_FOLDER='EX_file_list_folder'
11
13
 
12
14
  def assert_usage(assertion,errmsg)
13
15
  return if assertion
@@ -58,6 +60,11 @@ if transfer_spec.has_key?(TS_LOGLEVEL)
58
60
  Asperalm::Log.instance.level=transfer_spec[TS_LOGLEVEL]
59
61
  transfer_spec.delete(TS_LOGLEVEL)
60
62
  end
63
+ # possibly override temp folder
64
+ if transfer_spec.has_key?(TS_TMP_FILELIST_FOLDER)
65
+ Asperalm::Fasp::Parameters.file_list_folder=transfer_spec[TS_TMP_FILELIST_FOLDER]
66
+ transfer_spec.delete(TS_TMP_FILELIST_FOLDER)
67
+ end
61
68
  # get local agent (ascp)
62
69
  client=Asperalm::Fasp::Local.instance
63
70
  # disable ascp output on stdout to not mix with JSON events
@@ -1 +1 @@
1
- 0.10.3
1
+ 0.10.4
@@ -16,7 +16,7 @@ module Asperalm
16
16
  MAX_REDIRECT=10
17
17
  private_constant :VAL_ALL,:MAX_REDIRECT
18
18
  attr_reader :api_aoc
19
- attr_accessor :option_ak_secret
19
+ attr_accessor :option_ak_secret,:option_secrets
20
20
  def initialize(env)
21
21
  super(env)
22
22
  @default_workspace_id=nil
@@ -26,10 +26,12 @@ module Asperalm
26
26
  @home_node_file=nil
27
27
  @api_aoc=nil
28
28
  @option_ak_secret=nil
29
+ @option_secrets={}
29
30
  @url_token_data=nil
30
31
  @user_info=nil
31
32
  @ats=Ats.new(@agents.merge(skip_secret: true))
32
33
  self.options.set_obj_attr(:secret,self,:option_ak_secret)
34
+ self.options.set_obj_attr(:secrets,self,:option_secrets)
33
35
  self.options.add_opt_list(:auth,Oauth.auth_types,"type of Oauth authentication")
34
36
  self.options.add_opt_list(:operation,[:push,:pull],"client operation for transfers")
35
37
  self.options.add_opt_simple(:client_id,"API client identifier in application")
@@ -38,6 +40,7 @@ module Asperalm
38
40
  self.options.add_opt_simple(:private_key,"RSA private key PEM value for JWT (prefix file path with @val:@file:)")
39
41
  self.options.add_opt_simple(:workspace,"name of workspace")
40
42
  self.options.add_opt_simple(:secret,"access key secret for node")
43
+ self.options.add_opt_simple(:secrets,"access key secret for node")
41
44
  self.options.add_opt_simple(:eid,"identifier") # used ?
42
45
  self.options.add_opt_simple(:name,"resource name")
43
46
  self.options.add_opt_simple(:link,"public link to shared resource")
@@ -58,6 +61,8 @@ module Asperalm
58
61
  self.options.parse_options!
59
62
  return if env[:man_only]
60
63
  update_aoc_api
64
+ raise CliBadArgument,"secrets shall be Hash" unless @option_secrets.is_a?(Hash)
65
+ @api_aoc.add_secrets(@option_secrets)
61
66
  end
62
67
 
63
68
  def user_info
@@ -559,7 +564,7 @@ module Asperalm
559
564
  set_workspace_info
560
565
  set_home_node_file
561
566
  # set node secret in case it was provided
562
- @api_aoc.secrets[@home_node_file[:node_info]['id']]=@option_ak_secret
567
+ @api_aoc.add_secrets({@home_node_file[:node_info]['access_key']=>@option_ak_secret})
563
568
  command_repo=self.options.get_next_command(NODE4_COMMANDS.clone.concat([:short_link]))
564
569
  case command_repo
565
570
  when *NODE4_COMMANDS; return execute_node_gen4_command(command_repo,@home_node_file)
@@ -701,7 +706,7 @@ module Asperalm
701
706
  res_data=@api_aoc.read(resource_instance_path)[:data]
702
707
  # mandatory secret : we have only AK
703
708
  self.options.get_option(:secret,:mandatory)
704
- @api_aoc.secrets[res_data['id']]=@option_ak_secret unless @option_ak_secret.nil?
709
+ @api_aoc.add_secrets({res_data['access_key']=>@option_ak_secret}) unless @option_ak_secret.nil?
705
710
  api_node=@api_aoc.get_node_api(res_data)
706
711
  return Node.new(@agents.merge(skip_basic_auth_options: true, node_api: api_node)).execute_action if command.eql?(:v3)
707
712
  ak_data=api_node.call({:operation=>'GET',:subpath=>"access_keys/#{res_data['access_key']}",:headers=>{'Accept'=>'application/json'}})[:data]
@@ -5,13 +5,18 @@ require 'securerandom'
5
5
  require 'base64'
6
6
  require 'json'
7
7
  require 'securerandom'
8
+ require 'etc'
9
+ require 'fileutils'
8
10
 
9
11
  module Asperalm
10
12
  module Fasp
11
13
  # translate transfer specification to ascp parameter list
12
14
  class Parameters
13
15
  private
14
- @@file_list_folder=nil
16
+ # temp folder for file lists, must contain only file lists
17
+ # because of garbage collection takes any file there
18
+ # this could be refined, as , for instance, on macos, temp folder is already user specific
19
+ @@file_list_folder=File.join(Etc.systmpdir,Etc.getlogin)+'_asession_filelists'
15
20
  SEC_IN_DAY=86400
16
21
  # assume no transfer last longer than this
17
22
  # (garbage collect file list which were not deleted after transfer)
@@ -112,9 +117,9 @@ module Asperalm
112
117
  @builder.add_command_line_options(['--dest64'])
113
118
  end
114
119
 
115
- # use file list if there is storage defined for it.
116
120
  src_dst_list=@builder.process_param('paths',:get_value,:accepted_types=>Array,:mandatory=>!@job_spec.has_key?('keepalive'))
117
121
  unless src_dst_list.nil?
122
+ # use file list if there is storage defined for it.
118
123
  if @@file_list_folder.nil?
119
124
  # not safe for special characters ? (maybe not, depends on OS)
120
125
  Log.log.debug("placing source file list on command line (no file list file)")
@@ -122,6 +127,7 @@ module Asperalm
122
127
  else
123
128
  # safer option: file list
124
129
  # if there is destination in paths, then use filepairlist
130
+ # TODO: well, we test only the first one, but anyway it shall be consistent
125
131
  if src_dst_list.first.has_key?('destination')
126
132
  option='--file-pair-list'
127
133
  lines=src_dst_list.inject([]){|m,e|m.push(e['source'],e['destination']);m}
@@ -149,18 +155,20 @@ module Asperalm
149
155
  return env_args
150
156
  end
151
157
 
152
- # temp files are created here (if value is not nil)
153
- # garbage collect undeleted files
158
+ # temp file list files are created here
154
159
  def self.file_list_folder=(v)
155
160
  @@file_list_folder=v
156
- FileUtils.mkdir_p(@@file_list_folder)
157
- Dir.entries(@@file_list_folder).each do |name|
158
- file_path=File.join(@@file_list_folder,name)
159
- age_sec=(Time.now - File.stat(file_path).mtime).to_i
160
- # check age of file, delete too old
161
- if File.file?(file_path) and age_sec > FILE_LIST_AGE_MAX_SEC
162
- Log.log.debug("garbage collecting #{name}")
163
- File.delete(file_path)
161
+ unless @@file_list_folder.nil?
162
+ FileUtils.mkdir_p(@@file_list_folder)
163
+ # garbage collect undeleted files
164
+ Dir.entries(@@file_list_folder).each do |name|
165
+ file_path=File.join(@@file_list_folder,name)
166
+ age_sec=(Time.now - File.stat(file_path).mtime).to_i
167
+ # check age of file, delete too old
168
+ if File.file?(file_path) and age_sec > FILE_LIST_AGE_MAX_SEC
169
+ Log.log.debug("garbage collecting #{name}")
170
+ File.delete(file_path)
171
+ end
164
172
  end
165
173
  end
166
174
  end
@@ -82,10 +82,14 @@ module Asperalm
82
82
  def initialize(rest_params)
83
83
  super(rest_params)
84
84
  # access key secrets are provided out of band to get node api access
85
+ # key: access key
86
+ # value: associated secret
85
87
  @secrets={}
86
88
  end
87
89
 
88
- attr_reader :secrets
90
+ def add_secrets(secrets)
91
+ @secrets.merge!(secrets)
92
+ end
89
93
 
90
94
  # additional transfer spec (tags) for package information
91
95
  def self.package_tags(package_info,operation)
@@ -184,7 +188,7 @@ module Asperalm
184
188
  :base_url => node_info['url'],
185
189
  :headers => {'X-Aspera-AccessKey'=>node_info['access_key']},
186
190
  }
187
- ak_secret=@secrets[node_info['id']]
191
+ ak_secret=@secrets[node_info['access_key']]
188
192
  if ak_secret.nil? and node_scope.nil?
189
193
  raise 'There must be at least one of: secret, node scope'
190
194
  end
@@ -1,4 +1,6 @@
1
1
  require 'singleton'
2
+ require 'fileutils'
3
+
2
4
  module Asperalm
3
5
  # create a temp file name for a given folder
4
6
  # files can be deleted on process exit by calling cleanup
@@ -7,12 +9,14 @@ module Asperalm
7
9
  def initialize
8
10
  @created_files=[]
9
11
  end
12
+
10
13
  def temp_filelist_path(temp_folder)
11
14
  FileUtils::mkdir_p(temp_folder) unless Dir.exist?(temp_folder)
12
15
  new_file=File.join(temp_folder,SecureRandom.uuid)
13
16
  @created_files.push(new_file)
14
17
  return new_file
15
18
  end
19
+
16
20
  def cleanup
17
21
  @created_files.each do |filepath|
18
22
  File.delete(filepath)
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: asperalm
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.10.3
4
+ version: 0.10.4
5
5
  platform: ruby
6
6
  authors:
7
7
  - Laurent Martin
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-12-02 00:00:00.000000000 Z
11
+ date: 2019-12-04 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: xml-simple